Data monitoring system and electronic equipment

Abstract

The invention provides a data monitoring system and electronic equipment, and relates to the technical field of big data processing. The data monitoring system comprises a scene layer, an interface layer, a link layer and a data layer, and the scene layer is used for displaying a data index associated with a scene, a health index related to a service and data indicating a service state based on the scene corresponding to the service; the interface layer is used for managing a first file and interface change information associated with data exchange and monitoring data in an interface operation process, and the first file is used for indicating a demand rule associated with the data exchange; the link layer is used for displaying an execution sequence and a dependency relationship of tasks associated with a service, and supporting query of information of each task and a fault source; and the data layer is used for collecting logs and data indexes of all the components and carrying out fault positioning based on the logs of all the components. The abnormal source positioning efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data monitoring system and electronic device. Background Technology

[0002] In recent years, with the growth of data volume, big data platforms can be used to aggregate scattered data from various channels and process them systematically.

[0003] Currently, in big data platforms, the data processing from upstream data supply systems (such as data warehouses and external data sources) to downstream application systems involves multiple stages, including data collection, transmission, processing, storage, and application. Business personnel need to monitor the data processing status in real time to ensure that the data is accurate, complete, and timely.

[0004] However, if an anomaly occurs during data processing, business personnel may find it difficult to quickly pinpoint the source of the anomaly. Summary of the Invention

[0005] This application provides a data monitoring system and electronic equipment to improve the efficiency of anomaly source location.

[0006] In a first aspect, embodiments of this application provide a data monitoring system method, comprising: a scene layer, an interface layer, a link layer, and a data layer, wherein:

[0007] The scenario layer is used to display data metrics associated with the scenario, as well as business-related health metrics and data indicating business status, based on the scenario corresponding to the business.

[0008] The interface layer is used to manage the first file and interface change information associated with data exchange, and to monitor the data during the interface operation process. The first file is used to indicate the requirement rules associated with data exchange.

[0009] The link layer is used to display the execution order and dependencies of tasks related to business, and supports querying information about each task and the source of failure;

[0010] The data layer is used to collect logs and data metrics from each component and to locate faults based on the logs of each component.

[0011] In one possible implementation, the data monitoring system further includes a configuration layer, which is used for:

[0012] Configure the scenarios corresponding to the business, as well as the tasks required by the scenarios and the execution order and dependencies between the tasks;

[0013] Configure the data metrics that need to be monitored for the corresponding business scenarios, and configure the thresholds corresponding to the data metrics that need to be monitored;

[0014] Configure the alarm triggering conditions and alarm notification information for the corresponding business scenarios.

[0015] In one possible implementation, the interface layer is specifically used for:

[0016] Monitor data from multiple interfaces in the interface list, and mark any abnormal interfaces when they are identified; or,

[0017] Identify a list of interfaces associated with health metrics, and monitor data from multiple interfaces in the list associated with health metrics.

[0018] In one possible implementation, the link layer is specifically used for:

[0019] Obtain multiple nodes corresponding to the business scenario, where each node corresponds to a specific task required by the business scenario.

[0020] Configure the data metrics required for each task for each node, and determine the edges between each node based on the execution order and dependencies between multiple tasks in the business scenario to obtain the topology graph;

[0021] Based on the topology graph, the execution order and dependencies of tasks related to the business are displayed.

[0022] In one possible implementation, the link layer is also used for:

[0023] In the topology graph, the first faulty node is marked;

[0024] In response to the selection of the first node, information associated with the first node is displayed, wherein the information associated with the first node includes at least one of the following: the task type corresponding to the first node, the instance of the task, the triggering conditions of the task, and the data table information corresponding to the task.

[0025] In one possible implementation, the link layer is also used for:

[0026] Get the upstream and downstream nodes of the first node;

[0027] Based on the upstream and downstream nodes, the link that caused the first node to fail is identified, thus obtaining the source of the failure.

[0028] In one possible implementation, the scenarios corresponding to the business include data acquisition scenarios, data processing scenarios, and data supply scenarios.

[0029] In one possible implementation, the scenario corresponding to the service is a number-to-number scenario, wherein:

[0030] The data metrics associated with the data arrival scenario in the application layer include at least one of the following: data metrics associated with data arrival progress, data metrics associated with the non-arrival interface, data metrics associated with the data arrival interface, and data metrics associated with verification;

[0031] The data monitored during the interface operation process at the interface layer includes at least one of the following: arrival status, total number of files, actual arrivals within a historical period, verification status, number of files verified, number of files that failed verification, and verification pass rate.

[0032] In one possible implementation, the monitoring information related to file batches during the interface operation monitored by the interface layer includes at least one of the following:

[0033] The batch identifier, data arrival time, file size, number of file records, file path, check start time, check end time, number of failed check lines, and job status. Secondly, embodiments of this application provide an electronic device including the first aspect and / or various possible data monitoring systems described above.

[0034] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for implementing the first aspect and / or various possible data monitoring systems described above.

[0035] Fourthly, embodiments of this application provide a computer program product, including a computer program for implementing the first aspect and / or various possible data monitoring systems described above.

[0036] The data monitoring system and electronic equipment provided in this application embodiment, through a layered architecture design, present core data and health indicators focusing on business scenarios at the scenario layer, manage data exchange rules and monitor interface operation at the interface layer, visualize task execution order and dependencies at the link layer, and collect logs and locate faults at the data layer. This achieves end-to-end monitoring of the entire data processing process from business scenarios to the technical link, allowing business personnel to intuitively grasp the business status and providing technical personnel with the ability to visualize task dependencies, collect logs, and locate faults. It avoids fragmented cross-system queries and manual correlation analysis during anomaly troubleshooting, shortens the time for locating the source of anomalies, ensures the accuracy, integrity, and timeliness of data processing, and improves the efficiency of data monitoring and operation and maintenance. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1This application provides a schematic diagram of the architecture of a data monitoring system.

[0039] Figure 2 This is a schematic diagram of the architecture of another data monitoring system provided in an embodiment of this application;

[0040] Figure 3 A flowchart illustrating the data monitoring method provided in this application;

[0041] Figure 4 A schematic diagram illustrating the use case of the data monitoring method provided in this application;

[0042] Figure 5 This is a flowchart illustrating a method for displaying a topology map provided in an embodiment of this application;

[0043] Figure 6 A flowchart illustrating a method for displaying a first page, provided as an embodiment of this application;

[0044] Figure 7 This is a flowchart illustrating a data transfer scenario provided in an embodiment of this application.

[0045] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0048] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0049] First, let me explain the terms used in this application:

[0050] Big data platform: It is a comprehensive platform that integrates data collection, storage, processing and analysis functions. The big data platform is used to integrate and extract value from massive and multi-type (structured, semi-structured and unstructured) data.

[0051] Source system: refers to the core component / system that generates and collects raw data, including the source of data generated in various business scenarios. The source system is used to generate and provide raw data to other systems (such as big data platforms).

[0052] In recent years, with the growth of data volume, big data platforms can be used to aggregate scattered data from various channels and process them systematically.

[0053] Currently, in big data platforms, the data processing from upstream data supply systems (such as data warehouses and external data sources) to downstream application systems involves multiple stages, including data collection, transmission, processing, storage, and application. Business personnel need to monitor the data processing status in real time to ensure that the data is accurate, complete, and timely.

[0054] However, if an anomaly occurs during data processing, business personnel need to describe the business phenomenon to technical personnel, who then need to check the logs in the upstream data supply system, big data platform, and application system, and manually perform correlation analysis. This process is fragmented and time-consuming, and it is easy to miss key steps, making it difficult for business personnel to quickly locate the source of the anomaly.

[0055] To address the aforementioned issues, this application provides a data monitoring system comprising a scenario layer, an interface layer, a link layer, and a data layer. Specifically: the scenario layer displays scenario-related data metrics, business-related health metrics, and data indicating business status based on the business-specific scenario; the interface layer manages the first file and interface change information associated with data exchange and monitors data during interface operation; the first file indicates the requirement rules associated with data exchange; the link layer displays the execution order and dependencies of business-related tasks and supports querying information about each task and the source of any faults; and the data layer collects logs and data metrics from each component and performs fault location based on the logs of each component.

[0056] In the aforementioned data monitoring system, a layered architecture design is employed. The scenario layer focuses on presenting core data and health indicators for business scenarios; the interface layer manages data exchange rules and monitors interface operation; the link layer visualizes task execution order and dependencies; and the data layer collects logs and enables fault location. This achieves end-to-end monitoring of the entire data processing process, from business scenarios to the technical links. It allows business personnel to intuitively grasp the business status and provides technical personnel with the ability to visualize task dependencies, collect logs, and locate faults. This avoids fragmented cross-system queries and manual correlation analysis during anomaly troubleshooting, shortens the time for locating the source of anomalies, ensures the accuracy, integrity, and timeliness of data processing, and improves the efficiency of data monitoring and operation and maintenance.

[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0058] Figure 1 This is a schematic diagram of the architecture of a data monitoring system provided in an embodiment of this application. Please refer to it for details. Figure 1 The architecture of the data monitoring system includes a scenario layer, an interface layer, a link layer, and a data layer, among which:

[0059] The scenario layer is used to display data metrics associated with the scenario, as well as business-related health metrics and data indicating business status, based on the scenario corresponding to the business.

[0060] The interface layer is used to manage the first file and interface change information associated with data exchange, and to monitor the data during the interface operation process. The first file is used to indicate the requirement rules associated with data exchange.

[0061] The link layer is used to display the execution order and dependencies of tasks related to business, and supports querying information about each task and the source of failure;

[0062] The data layer is used to collect logs and data metrics from each component and to locate faults based on the logs of each component.

[0063] In one possible implementation, the business scenario refers to a complete set of data processing flows based on specific business objectives, including the entire business chain from data access to result output.

[0064] In some implementations, the business scenarios include data acquisition scenarios, data processing scenarios, and data supply scenarios.

[0065] In one possible implementation, the scenario layer can be used to display key data metrics (such as metric charts) based on different business scenarios (e.g., data acquisition, processing, and data supply scenarios). It can also provide historical data comparisons across different dimensions to predict potential business fluctuations, and offer a real-time monitoring dashboard to display health metrics and real-time data on business performance that are of interest to the business. Ensuring the real-time monitoring dashboard has fast response times and high data refresh rates is crucial for providing the most immediate business insights. Furthermore, it can enhance user interaction by allowing users to customize scenarios and related metrics.

[0066] In one possible implementation, the first file may include a data exchange interface file and a data integration file, which define the data format, processing method, time requirements, and target system requirements of the data table or file.

[0067] In some implementations, the interface layer is specifically used for: monitoring data from multiple interfaces in the interface list and marking abnormal interfaces when they are identified; or, determining a list of interfaces associated with health indicators and monitoring data from multiple interfaces in the list of interfaces associated with health indicators.

[0068] In one possible implementation, the interface layer can be used to manage interfaces; it can also provide the function of marking important interfaces for focused attention during interface monitoring; it can also provide interface change management, monitor the change history of interface definitions, and the impact of changes on data exchange; it can also store the data arrival status and verification status; it can also monitor real-time monitoring data of the interface requirement execution process, display real-time monitoring data at each stage of the data link, and provide planned data for comparison; it can also monitor data on the entire list or a specific list of interfaces through filtering, identify abnormal interfaces and mark them; and it can also locate specific interface lists and identify abnormal situations based on the indicator data displayed in the scenario layer.

[0069] In one possible implementation, the link layer can be used to display a topology diagram, showing task links and data lineage, comprehensively monitoring the status of each node, allowing users to obtain node information by clicking on nodes, providing bidirectional tracing functionality (tracing links from front to back and vice versa to identify fault sources and potential impacts), and adding automated testing capabilities to the link layer to periodically verify the integrity and availability of the links. Here, a node refers to a visual unit in the link layer used to map specific tasks within a business scenario. Each node uniquely corresponds to a business task and is the basic element constituting the topology diagram, containing core attributes such as task identifier and task type.

[0070] In one possible implementation, the data layer can be used to store log data and metric data, as well as to query and analyze the metric data and send it to the monitoring system for collection and display. It can also be used to provide log query functionality based on Elasticsearch. The data layer can include data from the source system.

[0071] In this embodiment, the data monitoring system includes a scenario layer, an interface layer, a link layer, and a data layer. The scenario layer displays data metrics associated with the business scenario, as well as business-related health metrics and data indicating business status. The interface layer manages the first file and interface change information associated with data exchange and monitors data during interface operation. The first file indicates the requirement rules associated with data exchange. The link layer displays the execution order and dependencies of tasks associated with the business and supports querying information about each task and the source of faults. The data layer collects logs and data metrics from each component and performs fault location based on the logs of each component.

[0072] In the aforementioned data monitoring system, a layered architecture design is employed. The scenario layer focuses on presenting core data and health indicators for business scenarios; the interface layer manages data exchange rules and monitors interface operation; the link layer visualizes task execution order and dependencies; and the data layer collects logs and enables fault location. This achieves end-to-end monitoring of the entire data processing process, from business scenarios to the technical links. It allows business personnel to intuitively grasp the business status and provides technical personnel with the ability to visualize task dependencies, collect logs, and locate faults. This avoids fragmented cross-system queries and manual correlation analysis during anomaly troubleshooting, shortens the time for locating the source of anomalies, ensures the accuracy, integrity, and timeliness of data processing, and improves the efficiency of data monitoring and operation and maintenance.

[0073] Based on the above embodiments, the link layer is specifically used for:

[0074] Obtain multiple nodes corresponding to the business scenario, where each node corresponds to a specific task required by that scenario; configure the data metrics required for each task for each node, and determine the edges between nodes based on the execution order and dependencies between the multiple tasks in the business scenario to obtain a topology graph; based on the topology graph, display the execution order and dependencies of tasks associated with the business.

[0075] In one possible implementation, the data metrics are pre-defined quantitative monitoring standards for the tasks corresponding to the nodes, used to evaluate the effectiveness of the task's running status and data quality, including task execution-related metrics and data result-related metrics.

[0076] In one possible implementation, execution order and dependency refer to the sequential execution logic and data flow association between multiple tasks in a business scenario.

[0077] In one possible implementation, an edge is a visual line connecting two nodes in the topology graph, used to intuitively present the execution order and dependencies between nodes. The dependency type (such as data dependency, execution dependency) can be labeled on the line. The topology graph is a visual business link map built based on nodes and edges, which graphically presents the distribution, execution order, and dependencies of all tasks in the business scenario in a panoramic view. It is the core display form of the link layer.

[0078] In this embodiment, the task nodes corresponding to the business scenario are obtained through the link layer, quantitative monitoring indicators for task execution and data results are configured for the nodes, and visual edges between nodes are determined based on the sequential execution logic and data flow association between tasks, and a topology graph is constructed. The distribution, execution order, and data / execution dependency relationships of all tasks are presented in a graphical map, which not only realizes the visualization and transparency of the business data processing link, allowing operation and maintenance personnel to quickly grasp the task association logic without checking the system one by one, but also provides clear evaluation standards for task running status and data quality through preset quantitative indicators, laying the foundation for subsequent fault identification and location. This effectively solves the problem of low operation and maintenance efficiency caused by unclear traditional data processing links and unintuitive task associations, and improves the monitorability and management convenience of the entire business data processing process.

[0079] Based on the above embodiments, the link layer is also used for:

[0080] In the topology diagram, the first faulty node is marked; in response to the selection of the first node, information associated with the first node is displayed, wherein the information associated with the first node includes at least one of the following: the task type corresponding to the first node, the instance of the task, the triggering condition of the task, and the data table information corresponding to the task.

[0081] In one possible implementation, the first node is the node in the topology graph that is detected to be running abnormally or malfunctioning. That is, the corresponding task does not meet the preset data indicators or does not execute normally, and it is the core object of troubleshooting.

[0082] In this embodiment, the first faulty node (a task node that does not meet preset data indicators or does not execute normally) is differentiated in the topology diagram by the link layer, and the task type, instance, triggering conditions, data table information and other related content are displayed in response to the node selection operation. This not only realizes the rapid visual location of faulty nodes and avoids the operation and maintenance personnel from missing abnormal objects in complex links, but also provides the core business and technical details of faulty nodes in real time. The key information required for fault diagnosis can be obtained without cross-system queries, which shortens the time from the discovery of fault to the initial location of the problem direction, improves the accuracy and efficiency of data processing link fault diagnosis, and effectively solves the problem of difficulty in diagnosing related technical faults.

[0083] Based on the above embodiments, the link layer is also used for:

[0084] Obtain the upstream and downstream nodes of the first node; based on the upstream and downstream nodes, determine the link that caused the first node to fail, and obtain the source of the failure.

[0085] In one possible implementation, upstream and downstream nodes are relative concepts defined by dependencies. An upstream node is a preceding node that provides data to the current node or triggers execution, while a downstream node is a following node that depends on the data or execution results of the current node.

[0086] In this embodiment, the upstream node (the preceding node that provides data or triggers execution) and downstream node (the following node that depends on the current node's data or execution result) of the first fault node are traced through the link layer. Based on the dependency relationship between nodes, the core link that caused the first node's failure is located, and the source of the failure is clarified. This not only breaks through the limitation of traditional fault investigation focusing only on a single node, but also realizes in-depth analysis from single-point failure to link source tracing. It can also accurately pinpoint the root cause of the failure (such as abnormal data of upstream nodes, interruption of link dependencies, etc.), avoid blindly investigating irrelevant upstream and downstream nodes, shorten the time for fault root cause location, improve the pertinence and efficiency of data processing link fault repair, and effectively ensure the continuity and stability of business data processing.

[0087] Based on the above embodiments, exemplarily, Figure 2 For a detailed schematic diagram of another data monitoring system provided in this application embodiment, please refer to [link / reference]. Figure 2 In addition to the scene layer, interface layer, link layer, and data layer, the data monitoring system also includes a configuration layer, which is used for:

[0088] Configure the scenarios corresponding to the business, the tasks required for the scenarios, and the execution order and dependencies between the tasks; configure the data metrics to be monitored for the scenarios corresponding to the business, and configure the thresholds corresponding to the data metrics to be monitored; configure the alarm triggering conditions for the scenarios corresponding to the business, and the alarm notification information.

[0089] In one possible implementation, configuring the detection data metrics refers to selecting key operational metrics (such as execution success rate) and data quality metrics (such as field integrity) for the business scenario and task, and setting thresholds for normal / abnormal metrics.

[0090] In one possible implementation, the alarm triggering condition refers to a preset rule for triggering alarm notifications. When the monitored indicators meet abnormal conditions (such as falling below a threshold) or the task fails, an alarm is automatically triggered.

[0091] In one possible implementation, alarm notification information refers to the notification content pushed to relevant personnel when an alarm is triggered, including the alarm scenario, fault node, abnormal indicators and handling suggestions, while configuring the notification method and recipients.

[0092] In one possible implementation, after all configurations are completed at the configuration layer, the configuration layer encapsulates the configuration data such as scenarios, tasks, metrics, and alarms into a standardized format and synchronizes it to the scenario layer, link layer, and data layer.

[0093] The link layer constructs a topology map based on configuration data, the data layer collects logs and indicator data according to the indicator configuration, and the scenario layer displays business data and health indicators according to the configuration, forming a full-link configuration linkage.

[0094] In this embodiment, business scenarios, tasks and dependencies, monitoring metrics and thresholds, alarm triggering conditions, and notification information are uniformly configured at the configuration layer. Standardized configuration data is synchronized to the scenario layer, link layer, and data layer, achieving end-to-end configuration linkage. The link layer constructs a topology map based on the configuration, the data layer collects data according to the configuration, and the scenario layer displays metrics according to the configuration. This ensures accurate matching between data monitoring logic and business needs, meets the differentiated monitoring requirements of different business scenarios through customized configuration, and standardizes and automates the monitoring process, eliminating the need for manual, scattered configuration at each level. Simultaneously, alarm configuration allows for rapid push of fault details and handling suggestions, enabling relevant personnel to respond to anomalies promptly. This effectively solves the problems of cumbersome traditional monitoring configuration, poor coordination between levels, and delayed anomaly response, improving the flexibility, adaptability, and operational efficiency of the data monitoring system.

[0095] Based on the above embodiments, the business scenario corresponds to a data-to-order scenario, wherein:

[0096] The data metrics associated with the data arrival scenario in the application layer include at least one of the following: data metrics associated with data arrival progress, data metrics associated with the non-arrival interface, data metrics associated with the data arrival interface, and data metrics associated with verification;

[0097] The data monitored during the interface operation process at the interface layer includes at least one of the following: arrival status, total number of files, actual arrivals within a historical period, verification status, number of files verified, number of files that failed verification, and verification pass rate.

[0098] In one possible implementation, the data status refers to the overall progress and result of data transmission from the upstream data supply system to the target system. It is used to indicate whether the data transmission process is completed normally, including "not started", "transmitting", "completed" and "transmission failed". The upstream data supply system is the source of the data. Its type can be flexibly configured according to actual business needs, such as data warehouses built within the enterprise, external data sources provided by third parties, and other common data supply carriers.

[0099] The total number of files refers to the total number of data files that the upstream data supply system plans to transmit under the current business scenario;

[0100] The actual number of data transmitted within a historical period refers to the actual amount of data successfully transmitted to the target system within a preset historical time window (such as the last hour or the last 24 hours). This amount of data can be measured by the number of files, data entries, or data size.

[0101] The verification status refers to the progress and final result of the data quality verification process for data that has been successfully delivered to the target system. It is used to indicate whether the delivered data meets the quality requirements, including "not verified", "verification in progress", "verification passed", "verification failed" and "partial verification passed".

[0102] The number of checked files refers to the total number of files for which data quality verification has been completed;

[0103] The number of files that failed the check refers to the total number of files that did not meet the rule requirements during data quality verification (such as missing fields, incorrect data format, or data values ​​exceeding the reasonable range).

[0104] The pass rate of the inspection refers to the percentage of data that has passed quality inspection.

[0105] In one possible implementation, the data metrics associated with the data acquisition scenarios in the application layer can be as shown in Table 1.

[0106] Table 1

[0107]

[0108] In some implementations, the monitoring information related to file batches during the interface operation process monitored by the interface layer includes at least one of the following:

[0109] Batch identifier, data arrival time, file size, number of records in the file, file path, check start time, check end time, number of lines that failed the check, and job status.

[0110] The batch identifier refers to a string generated by the system according to preset rules (such as concatenating date, serial number, business code and timestamp);

[0111] The time of data arrival refers to the timestamp when all files in the batch have been completely transferred to the target system (such as the receiving directory of a big data platform);

[0112] The file size refers to the total storage capacity of a single file or the entire file in this batch;

[0113] The file record count refers to the total number of valid data records contained in the batch of files;

[0114] The file path refers to the storage location of the batch of files in the target system;

[0115] The check start time refers to the timestamp at which the interface layer initiates the data quality check for this batch of files;

[0116] The end time of verification refers to the timestamp when the data quality verification of this batch of documents is completely completed (including verification of each record and statistical analysis of results);

[0117] The number of rows that failed the check refers to the number of records in this batch of files that did not meet the preset rules (such as incorrect field format, data values ​​exceeding the threshold, missing key fields, or failed logical checks) during data quality verification.

[0118] The status of a job refers to the overall operational status of the interface layer data transmission and verification job corresponding to that batch, including "Pending Transmission", "Transmission in Progress", "Transmission Failed", "Pending Verification", "Verification in Progress", "Verification Passed", "Verification Failed", and "Partially Passed".

[0119] Next, combined Figure 3 The data monitoring methods are explained. Figure 3 A flowchart illustrating the data monitoring method provided in this application is shown below. Figure 1 As shown, the method includes:

[0120] S301. Display the first page, which includes a first component. The first component displays the number of first interfaces that failed to process data. The first interfaces are associated with data requirements.

[0121] The execution subject of this application embodiment can be an electronic device or a data monitoring device installed in an electronic device. The data monitoring device can be implemented by software or by a combination of software and hardware.

[0122] In one possible implementation, the first page could be the management page of a big data platform.

[0123] In one possible implementation, the first interface includes data processing requirements and data processing configuration information.

[0124] In one possible implementation, data requirements may include: extracting data from the source system to the big data platform, processing the data in the big data platform, sending the data from the big data platform to the application system, and verifying the data in the big data platform.

[0125] In one possible implementation, the configuration information for data processing is used to indicate the data source, processing rules, verification rules, and / or execution time. For example, if the data requirement is to extract data from a table in a source system and transfer it to a big data platform, the configuration information for data processing may include the address of the source system, the path to the data storage, the data format (e.g., comma-separated values ​​(CSV)), processing rules (i.e., data filtering conditions, such as filtering data after October 2025), verification rules (e.g., the non-empty rate of the order amount field is ≥99%), and the execution time (e.g., 8 AM daily).

[0126] S302. In response to touch on the first component, display information of the first interface, the information of the first interface being used to indicate whether data processing corresponding to the first interface has been performed.

[0127] In one possible implementation, in response to a touch on the first component, information about the first interface can be displayed in the form of a list. Each record in the list corresponds to a first interface, and each record includes the following: the status of the first interface and the number of nodes that failed to execute and process data.

[0128] The status of the first interface can be "processing completed", "processing not completed", or "processing not completed". Each node can be a technical link involved in the data processing process associated with the first interface.

[0129] In one possible implementation, the status of the first interface is "processing completed", indicating that the first interface has completed data processing according to the execution time in its configuration information;

[0130] The status of the first interface is "processing not completed", indicating that the first interface has reached the execution time specified in its configuration information, but the data processing has not been completed. This may be due to node failure during data processing or interruption caused by dependent resources (such as data not being ready in the source system).

[0131] The status of the first interface is "Not yet processed and incomplete", indicating that the first interface has not reached its configured execution time and data processing has not been triggered at present.

[0132] S303. In response to a touch of the control corresponding to the executed first interface, a topology graph is displayed, the topology graph including the states of multiple nodes, the nodes being associated with the steps required for data processing corresponding to the executed first interface.

[0133] In one possible implementation, the node represents the technical step required for each step of data processing corresponding to the first interface.

[0134] For example, if the data requirement corresponding to the first interface is to extract data from the source system to the big data platform, then the nodes corresponding to the first interface may include: source system data reading node, data format verification node, data transmission node, big data platform data receiving node, and data integrity verification node. Each node will work together in sequence to realize the data processing corresponding to the first interface.

[0135] Among them, the source system data reading node is used to read data from the source system, the data format verification node is used to check the format of the read data, such as checking whether the number of fields and data types of the data match the preset rules, the data transmission node is used to transmit the verified data to the big data platform using the preset transmission protocol, the big data platform data receiving node is used to receive the transmitted data and temporarily store it in the temporary storage area of ​​the big data platform, and the data integrity verification node is used to check whether the amount of received data is consistent with the amount of data to be transmitted from the source system and to confirm that there is no packet loss.

[0136] The preset rules can include the number of fields and data types of the data. For example, if the first interface extracts data from the order table, the number of fields is 5 (fields include: order identifier, user identifier, product identifier, order time and order amount). The order identifier is of string type, the user identifier is of string type, the product identifier is of string type, the order time is of date and time type, and the order amount is of numeric type with 2 decimal places.

[0137] The preset transmission protocol can be a transmission protocol based on Transmission Control Protocol / Internet Protocol (TCP / IP). For example, it can be Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), or Kafka. The specific protocol selection can be determined according to the amount of data corresponding to the first interface and the timeliness requirements of transmission. For example, the first interface with high real-time requirements can use the Kafka protocol, and the batch offline data extraction interface can use the FTP protocol.

[0138] S304. In response to touch on a node in an abnormal state, display the reason why the first interface that was executed failed to process data.

[0139] In some implementations, the specific method for displaying the reason for the data processing failure of the first executed interface in response to touch on a node in an abnormal state is as follows:

[0140] In response to touch on a node in an abnormal state, the node's information and a second component are displayed. The node's information includes the state of the instance corresponding to the node, which indicates whether the steps required for data processing corresponding to the node have been executed. In response to touch on the second component, the node's log information is displayed. The log information includes the reason why the executed first interface failed to process data.

[0141] For example, the state of an instance is executing, executed, or not executed.

[0142] In one possible implementation, in addition to the instance's state, the node's information also includes the node name, node identifier, instance name, and business date.

[0143] In one possible implementation, nodes and instances are associated. A node is an abstract definition, while an instance is an entity generated by a node during a specific runtime. The node name can indicate the type of the node, such as "data check node". The type of the node identifier can be, for example, a string type, and the type of the instance name can be, for example, a string type. This application embodiment does not limit this.

[0144] In one possible implementation, log information may include node information, node metric data, and operation information of the instance corresponding to the node. The specific operation information can be used to analyze and locate the source and specific cause of the anomaly.

[0145] For example, the data requirement corresponding to the first interface is to extract data from the source system to the big data platform. If the data reading node of the source system is abnormal, the operation information of the instance corresponding to the node will be "Reason for data retrieval failure: Source system database connection timeout, failure time: time A" and other information.

[0146] If an anomaly occurs in the data format verification node, the operation information for the corresponding instance of the node will be messages such as "Incorrect data format transmission, field 'user identifier' type mismatch".

[0147] In one possible implementation, in response to a touch on the second component, log information of the node is displayed, including the reason why the node failed to process data, that is, the reason why the first interface that was executed failed to process data.

[0148] In this embodiment, a first page is displayed, including a first component. The first component displays the number of first interfaces that failed data processing, and each first interface is associated with a data requirement. In response to touch on the first component, information about the first interfaces is displayed, indicating whether the data processing corresponding to the first interface has been executed. In response to touch on the control corresponding to the executed first interface, a topology graph is displayed, including the status of multiple nodes, each node being associated with the steps required for the data processing corresponding to the executed first interface. In response to touch on a node in an abnormal state, the reason for the data processing failure of the executed first interface is displayed.

[0149] In the above method, by touching the first component, business personnel can check whether the data processing corresponding to the first interface has been executed, thereby quickly eliminating failures caused by non-execution, focusing on the core problem interfaces that have been executed but failed, and reducing ineffective troubleshooting actions. In addition, by displaying the topology diagram, the status of the data processing steps corresponding to the executed interfaces can be intuitively shown to business personnel, allowing them to quickly identify the abnormal nodes without having to browse through scattered logs. Furthermore, by touching the abnormal nodes, business personnel can view the specific reasons for the data processing failure (such as source system database connection timeout, data field type mismatch, insufficient transmission bandwidth, etc.), eliminating the intermediate step from identifying the abnormal node to deduce the specific reasons for the node's failure. In summary, business personnel can complete the anomaly location by performing a series of click operations on the big data platform page, thereby improving the efficiency of anomaly source location.

[0150] Based on the above embodiments, the following will be conducted through... Figure 4 The application scenarios of the data monitoring method provided in the embodiments of this application are described, for example, Figure 4 A schematic diagram illustrating the use case of the data monitoring method provided in this application.

[0151] like Figure 4 As shown in Figure a, the electronic device displays page 401, which is the interface provided by the big data platform (the first page in the above embodiment). Page 401 includes a first component 4011, on which the number of first interfaces that failed data processing is displayed as "4". In response to a touch on the first component 4011, the electronic device displays page 402.

[0152] like Figure 4 As shown in b, page 402 includes an information table 4021 and a control 4022 (the control corresponding to the executed first interface in the above embodiment). The information table 4021 includes information about the first interface. In response to a touch of the control 4022, the electronic device displays page 403.

[0153] like Figure 4As shown in c, page 403 includes a topology graph 4031, which includes the states of node A, node B (a node in an abnormal state), and node C. In response to a touch on node B, the electronic device displays page 404, which is a page that overlays information about node B and a second component 4041 on top of page 403.

[0154] like Figure 4 As shown in d, page 404 includes information about node B and a second component 4041. The information about node B includes the node B name, the node B identifier, the instance name corresponding to node B, and the business date. In response to a touch of the second component 4041, the electronic device displays page 405, which is a page overlaid on page 403 displaying the reason why node B failed to process data.

[0155] like Figure 4 As shown in e, page 405 includes the page explaining why node B failed to process data, which is the reason why the first interface that was executed failed to process data.

[0156] In this embodiment, business personnel can easily identify abnormal nodes and obtain the reasons for the abnormality by performing a series of touch operations on the big data platform, thereby improving the efficiency of locating the source of the abnormality.

[0157] Based on any of the above embodiments, the following, in conjunction with Figure 5 Detailed description of the topology diagram ( Figure 3 The method in S303 of the embodiment.

[0158] Figure 5 A flowchart illustrating a method for displaying a topology map provided in this application embodiment is shown below for details. Figure 5 The method may include:

[0159] S501, In response to touch on the control corresponding to the executed first interface, obtain the association information of multiple nodes corresponding to the first interface and the indicator data of multiple nodes.

[0160] In one possible implementation, the association information of multiple nodes corresponding to the first interface may include: the basic attributes of the nodes, the dependencies between nodes, and the business affiliation information.

[0161] In one possible implementation, the basic attributes of a node may include: the node's identifier, node name, node type (data read / transfer / check / store), and the corresponding technical component (e.g., a read node corresponds to the Flink Reader technical component, and a transfer node corresponds to the Kafka Producer technical component).

[0162] In one possible implementation, the dependencies between nodes may include: the identifier and dependency type of the direct upstream node (strong dependency / weak dependency, such as a data checking node having a strong dependency on a data receiving node, which cannot check if no data is received; a log recording node having a weak dependency on a data processing node, which does not affect the core process), and a list of fields of the dependent data (such as fields that the upstream node needs to pass to the downstream node, such as the order identifier and order time).

[0163] In one possible implementation, business attribution information may include: the first interface associated with the node, the business scenario corresponding to the first interface (such as "financial data processing"), and the business priority (high / medium / low).

[0164] In one possible implementation, the node's metrics may include: whether the node has completed execution, the node's execution time (e.g., if the expected 30 minutes but the actual execution time is 45 minutes, it is marked as abnormal), the number of node retries (if it fails after more than 3 preset retries, it is marked as abnormal), the amount of data processed (the deviation rate between the actual number of data entries processed and the expected number of data entries; if the deviation rate exceeds 10%, an alert is issued), the data transmission rate (if the transmission node's rate is lower than the preset transmission speed of 100MB / s, it is marked as abnormal), the data integrity (e.g., the field missing rate of received data; if it exceeds the preset accuracy threshold of 1%, it is marked as abnormal), the data accuracy (e.g., the percentage of erroneous data entries found by the check node; if it exceeds the preset percentage of 5%, it is marked as abnormal), and the data consistency (e.g., the matching rate of data verification between upstream and downstream nodes; if it is lower than the preset matching rate of 99.9%, it is marked as abnormal).

[0165] It should be noted that a more detailed explanation of the node's indicator data can be found in the subsequent implementation examples, and will not be repeated here.

[0166] S502. Based on the association information of multiple nodes corresponding to the first interface and the indicator data of multiple nodes, determine the topology graph. The association information is used to indicate the dependency relationship between multiple nodes.

[0167] In some implementations, the topology is determined based on the association information of multiple nodes corresponding to the first interface and the indicator data of multiple nodes, which may include:

[0168] Based on the association information of multiple nodes corresponding to the first interface, determine the first position of multiple nodes in the topology graph and the edge information of multiple nodes; based on the first position of multiple nodes in the topology graph and the edge information of multiple nodes, determine the topology graph; for any node among the multiple nodes, if the node's index data is not a safe value, mark the node's state as an abnormal state in the topology graph.

[0169] In one possible implementation, multiple nodes can be arranged first, and then the position of the arranged nodes in the topology can be determined as the first position. The multiple nodes corresponding to the sequentially executed data processing steps can be arranged horizontally, and the multiple nodes corresponding to the parallelly executed data processing steps can be arranged vertically.

[0170] Optionally, multiple parallel nodes can be enclosed in dashed boxes and labeled with the words "parallel processing" to clarify the parallel processing relationship between the multiple nodes.

[0171] In one possible implementation, the edge information can be associated with the data processing steps between nodes. For example, the edge information between nodes is "Transmission: Incremental data synchronization (based on timestamp filtering, only transmit data updated after time B)", indicating that the processing steps from the upstream node to the downstream node are to perform timestamp filtering on the full data and only synchronize the incrementally updated data to the downstream node.

[0172] In one possible implementation, the security value can be pre-set by business personnel on the big data platform page. For example, the node is a node that detects whether a field is not empty (the non-empty rate can be obtained), and the security value is any value between 99% and 100%. When the non-empty rate is 70%, the status of the node can be marked as abnormal.

[0173] In one possible implementation, the node's state can be marked as abnormal by marking the corresponding icon in red on the topology graph and / or displaying the word "abnormal" below the corresponding icon.

[0174] In some implementations, determining the first position of multiple nodes in the topology graph and the edge information of multiple nodes based on the association information of multiple nodes corresponding to the first interface can include:

[0175] Based on the association information of multiple nodes corresponding to the first interface, the order of steps required for data processing of multiple nodes and the dependencies between multiple nodes are determined; based on the order of steps required for data processing of multiple nodes, the second position of multiple nodes in the topology graph is determined, and the second position is used to indicate the initial arrangement of multiple nodes in the topology graph according to the order of steps required for data processing; based on the dependencies between multiple nodes, the second position of multiple nodes in the topology graph is adjusted to obtain the first position of multiple nodes in the topology graph; based on the steps required for data processing of multiple nodes, the edge information of multiple nodes is determined.

[0176] In one possible implementation, adjacent nodes in the second position are spaced the same in the topology graph.

[0177] In one possible implementation, when the dependencies between nodes are strong, the spacing between nodes can be reduced to obtain the first position of each node, and they can be connected with a bold solid line to highlight the core dependency relationship. When the dependencies between nodes are weak, the spacing between nodes can be increased or the default spacing can be maintained to obtain the first position of each node, and they can be connected with a thin dashed line. When a node depends on multiple upstream nodes, the horizontal position of the node can be adjusted to the middle position of the multiple upstream nodes, so that the length of the line connecting the node to the multiple upstream nodes is similar, and the first position of each node can be obtained. The lines can be distributed radially to avoid the lines crossing and tangling.

[0178] In one possible implementation, the steps required for data processing for multiple nodes can be taken, the edge information of multiple nodes can be determined, and the edge information can be marked on the connection line between multiple nodes.

[0179] S503, Display topology diagram.

[0180] One possible implementation could support full-screen or split-screen display of the topology map, and could also make the topology map adapt to different screen sizes.

[0181] In one possible implementation, the topology map can be determined at preset intervals and displayed on the big data platform's page. The preset interval is, for example, 3 seconds.

[0182] In this embodiment, the topology diagram can be used to show business personnel the data processing task chain and data lineage, and comprehensively monitor the status of each node. In addition, the dynamic updated topology diagram can intuitively show the chain status to business personnel. Furthermore, the topology diagram can provide bidirectional tracking functionality, enabling business personnel to trace the chain not only from front to back but also from back to front, in order to identify the source of anomalies and potential impacts (the source of anomalies can be identified by accessing upstream and downstream nodes with the node in an abnormal state as the center).

[0183] Based on any of the above embodiments, the following, in conjunction with Figure 6 Detailed description is displayed on the first page ( Figure 3 The method in S301 of the embodiment.

[0184] Figure 6 A flowchart illustrating a method for displaying a first page, as provided in this application embodiment, is available for detailed reference. Figure 6 The method may include:

[0185] S601, in response to the selection operation of a preset business scenario associated with the data processing requirement associated with the first interface, obtain the indicator data of multiple first interfaces and the number of first interfaces that failed data processing under the business scenario.

[0186] In some implementations, the indicator data includes indicator data generated during the data processing execution process corresponding to multiple first interfaces.

[0187] In one possible implementation, the preset business scenarios can be set by business personnel. These preset business scenarios include, for example, data acquisition scenarios, processing scenarios, and data supply scenarios. The data acquisition scenario can include extracting data from the source system to the big data platform or extracting data from the big data platform to the application system. The processing scenario can include processes such as cleaning, integrating, and calculating data on the big data platform. The data supply scenario can include monitoring the process of transmitting data from the source system to the big data platform or monitoring the process of the big data platform providing data to the application system.

[0188] In one possible implementation, if one of the multiple nodes of the first interface in the business scenario fails to process data, it indicates that the data processing corresponding to the first interface has failed.

[0189] In one possible implementation, the metric data for multiple first interfaces in a business scenario may include not only technical metric data but also business metric data, so that business personnel can analyze business conditions through visualized business metric data. In one possible implementation, the metrics for the first interfaces corresponding to each business scenario can be as shown in Table 2.

[0190] Table 2

[0191]

[0192] Among them, batch data link refers to the data flow link that processes static data accumulated to a certain level in a one-time, batch manner according to a preset period (such as hour, day, week); streaming data link refers to the data flow link that processes continuously generated dynamic data (such as user clicks, device logs) one by one; offline data link refers to the data flow link that extracts data offline from the source system to offline storage and then performs offline analysis and processing; real-time data link refers to the data flow link that supports instant response from data generation to processing completion and output of results.

[0193] S602. Perform structured processing on the indicator data to obtain at least one indicator graph. The indicator graph is used to indicate the indicator data generated during the data processing of multiple first interfaces in the business scenario.

[0194] In one possible implementation, the original indicator data can be cleaned and standardized before obtaining the indicator chart to avoid chart distortion due to data anomalies. For example, missing values, extreme values, and format errors can be filled or removed according to business rules. Different units of the same indicator can also be standardized, and the data period can be unified according to the time granularity of the business scenario.

[0195] In one possible implementation, the indicator chart may include: a biaxial line chart, a bar chart, a stacked bar chart, a scatter plot, a donut chart, and an area chart, etc.

[0196] S603, Load and display the indicator chart on the first page, and display the number of first interfaces that failed data processing on the first component of the first page.

[0197] In one possible implementation, at least one indicator chart can be displayed within a preset area on the first page.

[0198] In this embodiment, by associating a preset business scenario with the first interface, business personnel can obtain indicator data simply by selecting the relevant scenario, without having to manually filter through a massive number of interfaces. Furthermore, for different business scenarios and indicator characteristics of different links, exclusive indicator charts can be matched to clearly display data trends, differences, and proportions. In addition, by displaying the number of first interfaces that failed data processing on the first page, business personnel can see the anomalies as soon as they open the page, thereby enabling them to capture the anomalies immediately and avoid the backlog of problems.

[0199] Based on any of the above embodiments, the following, in conjunction with Figure 7 The process is explained in detail for several scenarios. Figure 7 This is a flowchart illustrating a data transfer scenario provided in an embodiment of this application. Please refer to it for details. Figure 7 .

[0200] Interface A can be obtained from the data requirement module, or parameter information can be obtained from an offline table. Based on this parameter information, interface B can be obtained through an online program (e.g., a Python program). Interfaces A and B are then transmitted to the supply module. The supply module can compare and analyze interfaces A and B. If interfaces A and B are consistent, data can be extracted from the source system and sent to the big data platform based on non-fungible tokens (NFTs) and the parameter information in interfaces A or B. The extracted data can then be statistically analyzed. In addition, interface B can be used to verify and statistically analyze the extracted data.

[0201] The data requirements module can include multiple interfaces, and the offline table can also include multiple interfaces.

[0202] The supply module is used to compare and analyze interface A and interface B to determine whether they are consistent. Interface A includes basic information such as the path of the data stored in the source system, the file name of the data in the source system, the frequency and initial data volume of the data transmitted from the source system to the big data platform (if it is incremental data transmission), and the path of the data stored in the big data platform.

[0203] Interface B includes technical information about the interface generated by the developer. For example, Interface B includes the path where the data is stored in the source system, the file name corresponding to the data in the source system, the frequency and initial data volume of the data transmitted from the source system to the big data platform (if it is incremental data transmission), the path where the data is stored in the big data platform, the verification method (e.g., comparing the number of records sent by the source system with the number of records received by the big data platform), and the data transmission protocol (e.g., HTTP).

[0204] In one possible implementation, data statistics include determining whether data has been transmitted to a big data platform, including the number of data files transmitted to the big data platform, the size of each file, and whether the data has been transmitted to a specified directory on the big data platform.

[0205] In one possible implementation, data verification and statistics include the number of records of statistical data, the size of statistical data, the results of a full comparison between the received data and the data sent by the source system, and the results of a sample comparison between the received data and the data sent by the source system. Based on the above verification and statistics results, it is determined whether the data has reached a preset pass rate (e.g., 90%). If the pass rate does not reach the preset pass rate, a warning message can be sent to business personnel. In addition, it can be determined whether the data transmission protocol is consistent with the data transmission protocol in interface B, where the data sent by the source system is the data corresponding to interface B in the source system.

[0206] In one possible implementation, data can be extracted from the source system using NFT, converted into a file, and then transferred to a specific path on the big data platform. Subsequently, based on business needs, the big data platform can load the stored file into a table for subsequent business use.

[0207] In one possible implementation, the results of data statistics and verification statistics can also be displayed on the first page of the big data platform.

[0208] In this embodiment, by comparing interface A in the data requirement module and interface B obtained through development, the problem of missing or incorrect interfaces caused by relying on a single module (such as relying solely on the data requirement module) can be avoided, thus solving the problem of a single interface source. In addition, by comparing and analyzing interface A and interface B through the supply module, the subsequent data processing stage is only entered when the two are consistent, which can intercept errors earlier. Furthermore, by checking and statistically analyzing, the problem of data loss during transmission can be located, enabling business personnel to judge data quality without professional technical skills.

[0209] This application also provides an electronic device, which includes the data monitoring system mentioned in the above embodiments.

[0210] This application also provides a computer program product, including a computer program that can be used to implement the above-described data monitoring system.

[0211] This application also provides a computer-readable storage medium storing computer-executable instructions, which can be used to implement the above-mentioned data monitoring system.

[0212] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0214] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0215] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0217] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the tasks of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0218] Those skilled in the art will understand that all or part of the tasks described in the above method embodiments can be implemented by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the tasks described in the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0219] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data monitoring system, characterized in that, It includes the scene layer, interface layer, link layer, and data layer, among which: The scenario layer is used to display data metrics associated with the scenario corresponding to the business, as well as health metrics and data indicating the business status related to the business. The interface layer is used to manage the first file and interface change information associated with data exchange, and to monitor the data during the interface operation process. The first file is used to indicate the requirement rules associated with data exchange. The link layer is used to display the execution order and dependencies of tasks associated with the business, and supports querying information about each task and the source of failure; The data layer is used to collect logs and data metrics from each component and to locate faults based on the logs of each component.

2. The data monitoring system according to claim 1, characterized in that, The data monitoring system also includes a configuration layer, which is used for: Configure the scenarios corresponding to the business, as well as the tasks required by the scenarios and the execution order and dependencies between the tasks; Configure the data metrics that need to be monitored for the corresponding business scenarios, and configure the thresholds corresponding to the data metrics that need to be monitored; Configure the alarm triggering conditions and alarm notification information for the corresponding business scenarios.

3. The data monitoring system according to claim 1, characterized in that, The interface layer is specifically used for: Monitor data from multiple interfaces in the interface list, and mark any abnormal interfaces when they are identified; or, Determine a list of interfaces associated with the health indicator, and monitor data from multiple interfaces in the list of interfaces associated with the health indicator.

4. The data monitoring system according to claim 1, characterized in that, The link layer is specifically used for: Obtain multiple nodes corresponding to the scenario of the business, wherein the multiple nodes correspond to multiple tasks required by the scenario of the business; Configure the data metrics required for each task for each node, and determine the edges between each node based on the execution order and dependencies between multiple tasks in the business scenario to obtain the topology graph; Based on the topology diagram, the execution order and dependencies of tasks related to the business are displayed.

5. The data monitoring system according to claim 4, characterized in that, The link layer is also used for: In the topology diagram, the first faulty node is marked; In response to the selection of the first node, information associated with the first node is displayed, wherein the information associated with the first node includes at least one of the following: the task type corresponding to the first node, the instance of the task, the triggering condition of the task, and the data table information corresponding to the task.

6. The data monitoring system according to claim 5, characterized in that, The link layer is also used for: Obtain the upstream and downstream nodes of the first node; Based on the upstream node and the downstream node, the link that caused the first node to fail is determined, and the source of the failure is obtained.

7. The data monitoring system according to any one of claims 1-6, characterized in that, The scenarios corresponding to the business include data acquisition scenarios, data processing scenarios, and data supply scenarios.

8. The data monitoring system according to claim 7, characterized in that, The scenario corresponding to the aforementioned business is a data-to-data scenario, wherein: The data metrics associated with the data arrival scenario in the application layer include at least one of the following: data metrics associated with data arrival progress, data metrics associated with the non-arrival interface, data metrics associated with the data arrival interface, and data metrics associated with verification; The data monitored by the interface layer during the interface operation process includes at least one of the following: arrival status, total number of files, actual arrivals within a historical period, verification status, number of files verified, number of files that failed verification, and verification pass rate.

9. The data monitoring system according to claim 8, characterized in that, The monitoring information related to file batches during the interface operation process monitored by the interface layer includes at least one of the following: Batch identifier, data arrival time, file size, number of records in the file, file path, check start time, check end time, number of lines that failed the check, and job status.

10. An electronic device, characterized in that, The electronic device includes the data monitoring system as described in any one of claims 1-9.

Images