Data full-link monitoring method, device, equipment and medium

Abstract

This application provides a data end-to-end monitoring method, apparatus, device, and medium. The method includes: first, responding to a data query request, generating a globally unique identifier based on the data query request; then, injecting and transmitting the globally unique identifier along the execution link of the data analysis request; next, based on the globally unique identifier, performing data association on monitoring data from different levels to generate end-to-end execution status data reflecting the data query request; and finally, performing anomaly detection and fault location on the end-to-end execution status data. This method eliminates the risk of data misalignment in high-concurrency scenarios associated with traditional timestamp-based association methods, significantly shortens fault location time, and thus improves operational efficiency and system stability.

Description

Technical Field

[0001] This application relates to the field of big data analytics technology, and in particular to a method, apparatus, device, and medium for full-link data monitoring. Background Technology

[0002] In the financial industry, with the increasing complexity of business operations and the explosive growth of data volume, big data interactive analytics engines have become a core infrastructure supporting financial decision-making, risk control, and business optimization. A typical financial data analytics platform employs a multi-layered architecture, including a user interaction layer, a business intelligence (BI) layer, a distributed computing layer, and an underlying storage layer. While this architecture enables parallel processing and storage of large-scale data, its business chains are extremely long and have complex dependencies; performance fluctuations or failures at any layer can trigger global service interruptions. Furthermore, the financial industry has stringent requirements for system stability and data traceability. Therefore, there is an urgent need for a solution that enables end-to-end observability, end-to-end fault tracing, and intelligent resource management.

[0003] In existing technologies, common financial data analysis platforms typically employ a four-layer architecture (user interaction layer, BI layer, computing layer, and storage layer), with each layer managed through independent monitoring tools. At the device level, traditional IT monitoring tools are primarily used to monitor hardware resource metrics such as server CPU utilization, memory usage, disk I / O, and network traffic. At the component level, the reliance is mainly on the built-in web UI interfaces of various data components (e.g., the interactive analysis engine web UI displays query lists and basic statistics, the HDFS web UI displays storage capacity and node status, and the BI system backend displays the number of user sessions). Regarding business continuity, availability and responsiveness of services are ensured through methods such as periodic probing, HTTP heartbeat detection, and port scanning.

[0004] However, existing solutions for monitoring data are scattered across different systems, lacking a unified correlation mechanism, resulting in low efficiency in fault location. Summary of the Invention

[0005] This application provides a data end-to-end monitoring method, apparatus, device, and medium to solve the problems in the prior art where monitoring data is scattered across different systems, lacks a unified correlation mechanism, and has low efficiency in fault location.

[0006] In a first aspect, embodiments of this application provide a data end-to-end monitoring method, including:

[0007] In response to a data query request, a globally unique identifier is generated based on the data query request;

[0008] The globally unique identifier is injected and passed along the execution chain of the data analysis request;

[0009] Based on the globally unique identifier, data association is performed on monitoring data from different levels to generate end-to-end execution status data reflecting the data query request;

[0010] Anomaly detection and fault location are performed on the entire execution status data.

[0011] In one possible implementation, the execution chain passes through at least the service portal layer, the business intelligence system layer, the interactive analysis engine computing layer, and the distributed file storage layer in sequence.

[0012] The step of injecting and passing the globally unique identifier along the execution chain of the data analysis request includes:

[0013] When the service portal layer receives the data query request, it injects the globally unique identifier into the request metadata;

[0014] When the business intelligence system layer initiates a computation call to the interactive analysis engine computation layer, the globally unique identifier is injected into the call context or session attribute;

[0015] When the interactive analysis engine's computation layer initiates data read / write access to the distributed file storage layer, the globally unique identifier is injected into the storage access context.

[0016] In one possible implementation, the monitoring data at different levels includes:

[0017] Request access logs and response time metrics for the service portal layer;

[0018] Report parsing logs, query orchestration logs, and error code indicators at the business intelligence system layer;

[0019] The interactive analysis engine's computational layer includes task scheduling logs, operator execution logs, resource usage metrics, and query phase time metrics.

[0020] The distributed file storage layer includes read / write access logs, data block location information, and read / write latency metrics.

[0021] In one possible implementation, the step of associating monitoring data from different levels based on the globally unique identifier to generate end-to-end execution status data reflecting the data query request includes:

[0022] The collected monitoring data is cleaned, structured, parsed, and standardized to obtain standardized event data.

[0023] Based on the globally unique identifier, the standardized event data is aggregated across layers and aligned in time sequence.

[0024] Based on the aggregated data, the full-link execution status data is generated, which includes processing stages, time consumption, and status information at each level.

[0025] In one possible implementation, the anomaly detection of the end-to-end execution state data includes:

[0026] Extract time series of performance metrics associated with at least one of the globally unique identifiers from the end-to-end execution status data;

[0027] The time series of the performance metrics are analyzed using a pre-trained machine learning model to identify abnormal patterns.

[0028] The pre-trained machine learning model is trained in the following way:

[0029] Acquire end-to-end execution status data within a historical time period as training samples;

[0030] The training samples are subjected to feature engineering to extract feature vectors including time series statistical features, query text features, and link association features based on the globally unique identifier.

[0031] The initial machine learning model is trained using the feature vectors to obtain the pre-trained machine learning model.

[0032] In one possible implementation, fault location of the end-to-end execution status data includes:

[0033] In response to detected anomaly information, an end-to-end execution link view corresponding to the anomaly event is obtained based on the globally unique identifier;

[0034] The stage time, error information, and resource consumption information of each level are compared and analyzed according to the execution sequence of the execution links in the end-to-end execution link view.

[0035] Identify the level that contributes the most to the end-to-end time consumption and / or the level that first produces an error as candidate failure points;

[0036] In the hierarchy corresponding to the candidate fault point, log details, tracking details and storage access details under the same global unique identifier are associated, and fault location results are output. The fault location results include at least the fault level, fault stage, fault type and information corresponding to the fault.

[0037] In one possible implementation, the step of generating a globally unique identifier based on a data query request in response to the data query request includes:

[0038] Obtain the request initiation information and query statement information corresponding to the data query request. The request initiation information includes at least one of the following: initiation time, initiating entity identifier, and session identifier.

[0039] A globally unique identifier is generated based on the request initiation information and the query statement information, and a mapping relationship is established between the globally unique identifier and the data query request. The mapping relationship is used to identify the consistency of the same data query request in the subsequent monitoring data collection, association and tracing process.

[0040] Secondly, embodiments of this application provide a data end-to-end monitoring device, comprising:

[0041] The first generation module is used to generate a globally unique identifier in response to a data query request.

[0042] The first processing module is used to inject and transmit the globally unique identifier along the execution link of the data analysis request;

[0043] The second generation module is used to perform data association on monitoring data from different levels based on the globally unique identifier, and generate end-to-end execution status data reflecting the data query request.

[0044] The second processing module is used to perform anomaly detection and fault location on the full-link execution status data.

[0045] In one possible implementation, the execution chain passes through at least the service portal layer, the business intelligence system layer, the interactive analysis engine computing layer, and the distributed file storage layer in sequence.

[0046] The first processing module is specifically used for:

[0047] When the service portal layer receives the data query request, it injects the globally unique identifier into the request metadata;

[0048] When the business intelligence system layer initiates a computation call to the interactive analysis engine computation layer, the globally unique identifier is injected into the call context or session attribute;

[0049] When the interactive analysis engine's computation layer initiates data read / write access to the distributed file storage layer, the globally unique identifier is injected into the storage access context.

[0050] In one possible implementation, the monitoring data at different levels includes:

[0051] Request access logs and response time metrics for the service portal layer;

[0052] Report parsing logs, query orchestration logs, and error code indicators at the business intelligence system layer;

[0053] The interactive analysis engine's computational layer includes task scheduling logs, operator execution logs, resource usage metrics, and query phase time metrics.

[0054] The distributed file storage layer includes read / write access logs, data block location information, and read / write latency metrics.

[0055] In one possible implementation, the second generation module is specifically used for:

[0056] The collected monitoring data is cleaned, structured, parsed, and standardized to obtain standardized event data.

[0057] Based on the globally unique identifier, the standardized event data is aggregated across layers and aligned in time sequence.

[0058] Based on the aggregated data, the full-link execution status data is generated, which includes processing stages, time consumption, and status information at each level.

[0059] In one possible implementation, the anomaly detection of the end-to-end execution state data includes:

[0060] Extract time series of performance metrics associated with at least one of the globally unique identifiers from the end-to-end execution status data;

[0061] The time series of the performance metrics are analyzed using a pre-trained machine learning model to identify abnormal patterns.

[0062] The pre-trained machine learning model is trained in the following way:

[0063] Acquire end-to-end execution status data within a historical time period as training samples;

[0064] The training samples are subjected to feature engineering to extract feature vectors including time series statistical features, query text features, and link association features based on the globally unique identifier.

[0065] The initial machine learning model is trained using the feature vectors to obtain the pre-trained machine learning model.

[0066] In one possible implementation, the second processing module is specifically used for:

[0067] In response to detected anomaly information, an end-to-end execution link view corresponding to the anomaly event is obtained based on the globally unique identifier;

[0068] The stage time, error information, and resource consumption information of each level are compared and analyzed according to the execution sequence of the execution links in the end-to-end execution link view.

[0069] Identify the level that contributes the most to the end-to-end time consumption and / or the level that first produces an error as candidate failure points;

[0070] In the hierarchy corresponding to the candidate fault point, log details, tracking details and storage access details under the same global unique identifier are associated, and fault location results are output. The fault location results include at least the fault level, fault stage, fault type and information corresponding to the fault.

[0071] In one possible implementation, the first generation module is specifically used for:

[0072] Obtain the request initiation information and query statement information corresponding to the data query request. The request initiation information includes at least one of the following: initiation time, initiating entity identifier, and session identifier.

[0073] A globally unique identifier is generated based on the request initiation information and the query statement information, and a mapping relationship is established between the globally unique identifier and the data query request. The mapping relationship is used to identify the consistency of the same data query request in the subsequent monitoring data collection, association and tracing process.

[0074] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0075] The memory stores computer-executed instructions;

[0076] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0077] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0078] This application provides a data end-to-end monitoring method, apparatus, device, and medium. The method includes: firstly, in response to a data query request, generating a globally unique identifier based on the data query request to avoid confusion of monitoring information between different query requests and ensure the accuracy of link tracing; subsequently, injecting and transmitting the globally unique identifier along the execution link of the data analysis request. The principle behind this is that when the data analysis request is transmitted between processing levels, the globally unique identifier is written into the request context, session attributes, or call metadata, so that the identifier persists with the request at each level. This achieves unified labeling of log data, tracking data, and performance indicator data generated by the same data query request at different processing levels, thereby breaking down information barriers between levels. The process involves several steps. First, based on globally unique identifiers, data association is performed on monitoring data from different levels. This involves using globally unique identifiers as association keys to clean, structure, and align the monitoring data collected from each level. This results in the construction of end-to-end execution status data that reflects the entire data query request process. This integrates the monitoring information scattered across different levels into a holistic view that intuitively reflects the execution path, stage duration, and operational status. Finally, anomaly detection and fault location are performed on the end-to-end execution status data to enable rapid discovery and accurate location of anomalies in data query requests. This significantly improves the observability and fault handling efficiency of the data analysis system. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0080] Figure 1 Flowchart of the data end-to-end monitoring method provided in the embodiments of this application Figure 1 ;

[0081] Figure 2 Flowchart of the data end-to-end monitoring method provided in the embodiments of this application Figure 2 ;

[0082] Figure 3 This is a schematic diagram of the structure of the data end-to-end monitoring device provided in the embodiments of this application;

[0083] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0084] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0085] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0086] In the financial industry, with the increasing complexity of business operations and the explosive growth of data volume, big data interactive analytics engines have become the core infrastructure supporting financial decision-making, risk control, and business optimization. A typical financial data analytics platform employs a multi-layered architecture, including a user interaction layer (data analytics service portal), a business intelligence (BI) layer, a distributed computing layer (such as interactive analytics engines with MPP architectures like Trino), and an underlying storage layer (such as an HDFS cluster). While this architecture enables parallel processing and storage of large-scale data, its business chains are extremely long and its dependencies are complex. Performance fluctuations or failures at any layer can trigger a global service interruption. For example, when a user submits a query request for a critical risk control report through the BI system, the request must go through multiple stages, including SQL parsing, multi-node collaborative execution by the distributed computing engine, and data reading from HDFS, before finally returning the result. If any stage experiences a resource bottleneck or anomaly, traditional monitoring methods struggle to quickly pinpoint the root cause, leading to delays in business decisions or even the risk of data loss. In addition, the financial industry has strict requirements for system stability and data traceability. Existing technologies have significant shortcomings in cross-system data correlation, quantification of fault impact from a business perspective, and intelligent early warning. There is an urgent need for a solution that can achieve end-to-end observability, end-to-end fault tracing, and intelligent resource management.

[0087] In existing technologies, common financial data analysis platforms typically employ a four-layer architecture (user interaction layer, BI layer, computing layer, and storage layer), with each layer managed through independent monitoring tools. At the device level, traditional IT monitoring tools are primarily used to monitor hardware resource metrics such as server CPU utilization, memory usage, disk I / O, and network traffic. At the component level, the reliance is mainly on the built-in web UI interfaces of various data components (e.g., the interactive analysis engine web UI displays query lists and basic statistics, the HDFS web UI displays storage capacity and node status, and the BI system backend displays the number of user sessions). Regarding business continuity, availability and responsiveness of services are ensured through methods such as periodic probing, HTTP heartbeat detection, and port scanning.

[0088] However, existing solutions for monitoring data are scattered across different systems and lack a unified correlation mechanism. This lack of a unified correlation mechanism makes it impossible to quickly determine the cause of the fault, and technical indicators (such as CPU and memory) are out of touch with business scenarios, failing to intuitively reflect the impact of the fault on key reports or SQL queries.

[0089] To address the aforementioned problems, the inventors recognized that by establishing a unified identifier for the same data query request across the end-to-end execution chain, and using this identifier as a common anchor point for the association of monitoring data from different systems, it is possible to logically connect scattered monitoring data and achieve end-to-end visualized monitoring from business requests to basic resources. Based on this principle, this invention proposes a data end-to-end monitoring method, specifically: upon receiving a data query request, a globally unique identifier is generated, and this globally unique identifier is injected and transmitted along the execution chain of the data analysis request in the service portal layer, business intelligence system layer, interactive analysis engine computing layer, and distributed file storage layer; when monitoring data such as logs, traces, and metrics are generated at each level, the globally unique identifier is written into the corresponding monitoring data; subsequently, based on this globally unique identifier, the monitoring data from different levels are uniformly associated to construct end-to-end execution status data reflecting the end-to-end execution process of the data query request; and on this basis, anomaly detection and fault location are performed on the end-to-end execution status data. The above technical solution enables cross-system monitoring data fusion centered on specific data query requests, establishing a one-to-one correspondence between technical indicators and business query scenarios. This not only allows for the rapid identification of the specific level and stage of performance bottlenecks or faults, but also enables an intuitive assessment of the impact of anomalies on key reports or structured query statements, thereby significantly improving troubleshooting efficiency and system operational observability.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 1 Flowchart of the data end-to-end monitoring method provided in the embodiments of this application Figure 1 ;like Figure 1 As shown, the method includes:

[0092] S101. In response to a data query request, generate a globally unique identifier based on the data query request.

[0093] In one possible approach, the request initiation information and query statement information corresponding to the data query request are first obtained; then, a globally unique identifier is generated based on the request initiation information and query statement information, and a mapping relationship is established between the globally unique identifier and the data query request.

[0094] The mapping relationship is used to ensure consistency in identifying the same data query request during subsequent monitoring data collection, association and tracing. The request initiation information includes at least one of the following: initiation time, initiating entity identifier and session identifier.

[0095] It should be understood that, to achieve accurate end-to-end tracking of a single data query request, this application establishes a globally unique identifier at the request entry point. This is achieved by extracting and fusing source and content features that uniquely characterize the request, generating a highly unique and associative identifier (i.e., a globally unique identifier, UUID), and then strongly binding this identifier to the request. In specific implementation, when the service portal layer receives a user-submitted data query request (such as an SQL statement), it first obtains the request initiation information and query statement information. The request initiation information identifies "who" initiated the request "when"; the query statement information comes directly from the request content itself. Subsequently, based on the request initiation information and query statement information, a globally unique identifier is generated through hash calculation, encoding concatenation, or a combination of timestamps and random numbers. For example, "timestamp + user ID + hash value of the SQL statement" can be concatenated and hashed again to generate a string as the UUID. After generation, a mapping relationship is established between this globally unique identifier and the data query request, and stored in the context or cache. This mapping relationship is the cornerstone for the correct association of all subsequent monitoring data. It ensures that in the subsequent monitoring data collection, association and tracing process, the system can consistently identify all scattered data originating from the same data query request based on this UUID.

[0096] Understandably, by introducing a globally unique identifier during the data query request generation stage, precise identification and unified management of data query requests are achieved, laying the foundation for the automatic association and tracking of subsequent cross-level monitoring data, and avoiding the problem of monitoring data confusion caused by concurrent requests, retries, or complex links.

[0097] For example, suppose data analyst A submits a risk control report query SQL through session "Session_XYZ" at 10:00 AM. The system extracts "10:00", "User_A", "Session_XYZ", and the SQL text, generating a UUID such as "Req_9a3b8c7d6e5f". Subsequently, regardless of what logs or metrics this request generates in the BI system, interactive analytics engine, or HDFS, as long as it carries "Req_9a3b8c7d6e5f", the system can recognize that they belong to the same query.

[0098] S102. Inject and pass the globally unique identifier along the execution chain of the data analysis request.

[0099] The execution chain passes through at least the service portal layer, the business intelligence system layer, the interactive analysis engine computing layer, and the distributed file storage layer in sequence.

[0100] In one possible approach, when a data query request is received at the service portal layer, a globally unique identifier is first injected into the request metadata; then, when the business intelligence system layer initiates a computation call to the interactive analytics engine computation layer, the globally unique identifier is injected into the call context or session attribute; and finally, when the interactive analytics engine computation layer initiates a data read / write access to the distributed file storage layer, the globally unique identifier is injected into the storage access context.

[0101] It should be understood that this step aims to ensure that the UUID generated in S101 is completely copied and embedded into each subsequent processing stage, much like a gene, as the request processing flow progresses. Its implementation closely follows the data processing path that the execution chain passes through at least the service portal layer, the business intelligence (BI) system layer, the interactive analytics engine computation layer, and the distributed file storage layer. Specifically: First, when the service portal layer receives a data query request, the system injects the generated globally unique identifier into the request's metadata (such as HTTP headers or RPC call parameters), making it part of the request and passing it downstream. Then, when the business intelligence (BI) system layer processes the request and needs to initiate a computation call to the interactive analytics engine computation layer, the BI system extracts the UUID from the request metadata and injects it into the context or session attributes of this remote call, ensuring that the computation engine can recognize this identifier. Finally, when the interactive analytics engine computation layer executes a query and needs to initiate data read / write access to the underlying distributed file storage layer (such as HDFS), the engine continues to inject the UUID into the storage access context or the file read request's metadata. Through this series of proactive and coherent injection operations, the UUID is deeply embedded in the processing context of every key component through which the request flows.

[0102] For example, continuing from the example in S101, a request carrying the UUID "Req_9a3b8c7d6e5f" arrives at the BI system from the service portal. When the BI system calls the Trino engine after generating the execution plan, it will pass this UUID as a parameter in the RPC call. When the Trino engine executes and needs to read data blocks from HDFS, it will include this UUID in the read request sent to the HDFS DataNode.

[0103] Understandably, by continuously injecting and passing globally unique identifiers in the execution chain, contextual continuity across system calls is achieved, enabling data query requests in a distributed environment to have a complete, continuous, and traceable execution path.

[0104] S103. Based on a globally unique identifier, perform data association on monitoring data from different levels to generate end-to-end execution status data that reflects data query requests.

[0105] It should be noted that the monitoring data at different levels includes: request access logs and response time metrics at the service portal layer; report parsing logs, query orchestration logs, and error code metrics at the business intelligence system layer; task scheduling logs, operator execution logs, resource usage metrics, and query phase time metrics at the interactive analysis engine computing layer; and read / write access logs, data block location information, and read / write latency metrics at the distributed file storage layer.

[0106] In one feasible approach, the collected monitoring data is first cleaned, structured, parsed, and standardized to obtain standardized event data. Then, the standardized event data is aggregated across layers and aligned in time sequence based on a globally unique identifier. Finally, based on the aggregated data, full-link execution status data containing processing stages, time consumption, and status information at each level is generated.

[0107] It should be understood that this embodiment aims to solve the problem of isolated metrics. The principle behind this is to use the UUID injected in S102 as a key to connect various data along the execution chain, forming complete end-to-end execution status data. The monitoring data that needs to be associated at different levels mainly includes: request access logs and response time metrics at the service portal layer; report parsing logs, query orchestration logs, and error code metrics at the business intelligence (BI) system layer; task scheduling logs, operator execution logs, resource usage metrics, and query stage time metrics at the interactive analysis engine computing layer; and read / write access logs, data block location information, and read / write latency metrics at the distributed file storage layer. In specific implementation, firstly, the raw monitoring data asynchronously collected through the Agent or SDK is cleaned, structured, and standardized. For example, unstructured log text is parsed into structured events containing fields such as timestamps, levels, components, events, and UUIDs. The metric data is then standardized in unit and format to obtain standardized event data. Then, based on the common key of the globally unique identifier, standardized event data from different sources and in different formats are aggregated across layers and aligned in time sequence. That is, all events carrying the same UUID are sorted according to their timestamps and logically pieced together into a complete trajectory from the portal to the storage. Finally, based on the aggregated data, structured end-to-end execution status data is generated. This dataset contains the processing stages of this request at each level (such as "BI-SQL generation", "Trino-Stage-1", "HDFS-Read-Block-A"), the time spent at each stage, and key status information (such as success, failure, error code, and peak resource usage).

[0108] Understandably, by unifying and aggregating cross-level monitoring data, the system achieves full-link visualization of the execution status of data query requests, thereby improving the transparency and analyzability of the system for complex query execution processes.

[0109] It's also worth noting that after generating end-to-end execution status data reflecting the data query request, this data needs to be written in real-time to a real-time lake warehouse (such as Apache Doris) based on an MPP (Massively Parallel Processing) architecture. This lake warehouse organizes and compresses the data using a columnar storage format and partitions and sorts it based on data timestamps and globally unique identifiers. Through a real-time synchronization connector between the message queue and the lake warehouse, data flows continuously and stably into the warehouse. This approach constructs a unified, high-performance monitoring data hub, providing millisecond-level response capabilities for subsequent real-time queries and analysis. Its powerful parallel computing capabilities also support rapid correlation analysis and mining of massive historical monitoring data. Thus, all the scattered monitoring information originating from a single data query request and spanning multiple systems is ultimately integrated, correlated, and persisted in a unified data view that can be efficiently retrieved using globally unique identifiers.

[0110] It should be noted that the detailed process of anomaly detection and fault location for the entire execution status data is described in this application. Figure 2 The embodiments are described in detail, and the embodiments of this application will not be repeated here.

[0111] Understandably, by using anomaly detection and fault location based on end-to-end execution status data, the efficiency of locating data query problems has been significantly improved, the cost of manual troubleshooting has been reduced, and reliable data support has been provided for system performance optimization and stability assurance.

[0112] This application provides a data end-to-end monitoring method. The method includes: firstly, in response to a data query request, generating a globally unique identifier based on the data query request to avoid confusion of monitoring information between different query requests and ensure the accuracy of link tracing; subsequently, injecting and transmitting the globally unique identifier along the execution link of the data analysis request. The principle behind this method is that when the data analysis request is transmitted between different processing levels, the globally unique identifier is written into the request context, session attributes, or call metadata, so that the identifier persists with the request at each level. This achieves unified labeling of log data, trace data, and performance indicator data generated by the same data query request at different processing levels, thereby breaking down information barriers between levels. The process involves several steps. First, based on globally unique identifiers, data association is performed on monitoring data from different levels. This involves using globally unique identifiers as association keys to clean, structure, and align the monitoring data collected from each level. This results in the construction of end-to-end execution status data that reflects the entire data query request process. This integrates the monitoring information scattered across different levels into a holistic view that intuitively reflects the execution path, stage duration, and operational status. Finally, anomaly detection and fault location are performed on the end-to-end execution status data to enable rapid discovery and accurate location of anomalies in data query requests. This significantly improves the observability and fault handling efficiency of the data analysis system.

[0113] Figure 2 Flowchart of the data end-to-end monitoring method provided in the embodiments of this application Figure 2 ;like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the process of fault location for end-to-end execution status data is described in detail. This method includes:

[0114] S201. In response to the detected abnormal information, obtain the end-to-end execution link view corresponding to the abnormal event based on the globally unique identifier.

[0115] It should be noted that the process of anomaly detection for the end-to-end execution status data is as follows:

[0116] First, performance metric time series associated with at least one globally unique identifier are extracted from the end-to-end execution status data. Then, a pre-trained machine learning model is used to analyze the performance metric time series to identify abnormal patterns.

[0117] The pre-trained machine learning model is trained in the following way:

[0118] First, acquire the full-link execution status data within a historical time period as training samples; then, perform feature engineering on the training samples to extract feature vectors, including time series statistical features, query text features, and link association features based on globally unique identifiers; finally, use the feature vectors to train the initial machine learning model to obtain a pre-trained machine learning model.

[0119] It should be understood that by binding anomaly detection results to the entire execution chain through globally unique identifiers, anomaly analysis is no longer limited to a single system or a single metric, but can trace the contextual position of the anomaly within the entire execution path. In specific implementation, firstly, performance indicator time series associated with at least one globally unique identifier are extracted from the full-chain execution status data (such as the refresh time series of a BI report, or the read latency series of a HDFS data block); then, pre-trained machine learning models are used to analyze these performance indicator time series to identify complex anomaly patterns that are difficult to capture with static thresholds, such as the coordinated drift of multiple indicators, the disruption of periodic patterns, or slow trend deterioration.

[0120] The training process of the pre-trained machine learning model is as follows: First, acquire full-link execution status data under normal and known abnormal scenarios within a historical time period (such as the past three months) as training samples; then, perform in-depth feature engineering on the training samples to extract multi-dimensional feature vectors, including time series statistical features (such as mean, variance, and trend), query text features (obtained through SQL parsing and vectorization), and link association features based on globally unique identifiers (such as the correlation of upstream and downstream component indicators); finally, use these feature vectors to train and optimize the selected initial machine learning model (such as LSTM, Isolation Forest, or ensemble model) to obtain a pre-trained model that can accurately distinguish between normal and abnormal patterns.

[0121] Understandably, by introducing a machine learning model trained on historical data across the entire supply chain, intelligent identification of complex and hidden anomaly patterns is achieved, transforming anomaly detection capabilities from rule-driven to data-driven, and providing reliable triggering conditions for subsequent accurate fault location.

[0122] S202. Compare and analyze the stage time, error information and resource usage information of each level according to the execution sequence of the execution link in the end-to-end execution link view.

[0123] It should be understood that after obtaining the end-to-end execution chain view corresponding to the abnormal event, the system compares and analyzes the stage time, error information and resource consumption information of each level according to the order of execution chain. That is, by maintaining the consistency of execution order, the impact of the abnormality is unfolded in the time dimension and the chain dimension, thereby identifying the propagation path of the abnormality in the execution process.

[0124] In a specific implementation, the system analyzes the stage execution data of the service portal layer, business intelligence system layer, interactive analysis engine computing layer, and distributed file storage layer sequentially based on an end-to-end execution chain view. For each layer, the system calculates the percentage change of its stage time in the overall end-to-end time and makes a horizontal comparison with error code distribution and sudden increases or decreases in resource consumption. Simultaneously, the system can compare the execution characteristics of the current abnormal chain with similar normal chains to determine whether there is any abnormal amplification or triggering behavior at that layer.

[0125] Understandably, by using multi-dimensional comparative analysis based on execution order, the anomaly analysis process has a clear logical path, avoiding the blindness of traditional layer-by-layer investigation and improving the efficiency and accuracy of anomaly localization.

[0126] S203. Identify the level that contributes the most to the end-to-end time consumption and / or the level that first produces an error as candidate failure points.

[0127] It should be understood that after completing the comparative analysis of execution data at each level, the system further identifies the level that contributes the most to the end-to-end execution time and / or the level that first produces errors as candidate fault points. That is, by quantifying the impact of each level on the overall anomaly, the scope of fault investigation is narrowed, and the location target is concentrated on the level with the highest probability of a problem.

[0128] In a specific implementation, based on the aforementioned comparative analysis results, the system comprehensively evaluates the time increment, error occurrence time, and resource anomaly degree at each level. On one hand, the system identifies the level with the largest increase in time consumption ratio in the abnormal link; on the other hand, the system identifies the level where the error or abnormal indicator deviation first occurs in the execution link. When the above two types of levels overlap, they are directly identified as candidate failure points; when they do not overlap, both can be included in the candidate range and sorted according to the weight of the anomaly impact.

[0129] Understandably, by using a candidate fault point screening mechanism, the potential fault range in complex links can be significantly compressed, providing a clear target for subsequent refined analysis and reducing ineffective troubleshooting steps.

[0130] S204. In the hierarchy corresponding to the candidate fault point, associate the log details, trace details and storage access details under the same globally unique identifier, and output the fault location result.

[0131] The fault location results include at least the fault level, fault stage, fault type, and information corresponding to the fault.

[0132] It should be understood that after identifying candidate fault points, the system further associates log details, trace details, and storage access details under the same globally unique identifier at the level corresponding to the candidate fault point in order to output the final fault location result.

[0133] In a specific implementation, the system retrieves corresponding log records, call tracing data, and underlying storage access details from candidate fault levels based on globally unique identifiers, and performs time alignment and contextual correlation analysis on them. By analyzing abnormal keywords in the logs, breakpoint locations in the call chain, and abnormal delays or failure records in storage access, the system ultimately outputs structured fault location results.

[0134] Understandably, by performing refined data correlation within the candidate level, a closed-loop localization from anomaly detection to fault location is achieved, significantly improving fault handling efficiency and providing high-quality input for automated operation and maintenance and root cause analysis.

[0135] Figure 3 This is a schematic diagram of the structure of the data end-to-end monitoring device provided in the embodiments of this application; as shown below. Figure 3 As shown, the device includes:

[0136] The first generation module 301 is used to generate a globally unique identifier in response to a data query request.

[0137] The first processing module 302 is used to inject and pass the globally unique identifier along the execution link of the data analysis request;

[0138] The second generation module 303 is used to perform data association on monitoring data from different levels based on a globally unique identifier, and generate end-to-end execution status data that reflects the data query request.

[0139] The second processing module 304 is used to perform anomaly detection and fault location on the execution status data of the entire link.

[0140] In one possible implementation, the execution chain passes through at least the service portal layer, the business intelligence system layer, the interactive analysis engine computing layer, and the distributed file storage layer in sequence.

[0141] The first processing module 302 is specifically used for:

[0142] When receiving a data query request at the service portal layer, a globally unique identifier is injected into the request metadata;

[0143] When the business intelligence system layer initiates a computation call to the interactive analytics engine computation layer, a globally unique identifier is injected into the call context or session attribute;

[0144] When the interactive analytics engine's computation layer initiates data read / write access to the distributed file storage layer, a globally unique identifier is injected into the storage access context.

[0145] In one possible implementation, the monitoring data at different levels includes:

[0146] Request access logs and response time metrics for the service portal layer;

[0147] Report parsing logs, query orchestration logs, and error code indicators at the business intelligence system layer;

[0148] The interactive analysis engine's computational layer includes task scheduling logs, operator execution logs, resource usage metrics, and query phase time metrics.

[0149] The distributed file storage layer includes read / write access logs, data block location information, and read / write latency metrics.

[0150] In one possible implementation, the second generation module 303 is specifically used for:

[0151] The collected monitoring data is cleaned, structured, parsed, and standardized to obtain standardized event data.

[0152] Standardized event data is aggregated across layers and aligned in time series based on globally unique identifiers;

[0153] Based on the aggregated data, full-link execution status data is generated, which includes information on each level of processing, time consumption, and status.

[0154] In one possible implementation, anomaly detection is performed on the end-to-end execution state data, including:

[0155] Extract time series of performance metrics associated with at least one globally unique identifier from the end-to-end execution status data;

[0156] Utilize pre-trained machine learning models to analyze time series of performance metrics in order to identify anomalous patterns;

[0157] The pre-trained machine learning model is trained in the following way:

[0158] Acquire end-to-end execution status data within a historical time period as training samples;

[0159] The training samples are subjected to feature engineering to extract feature vectors, including time series statistical features, query text features, and link association features based on globally unique identifiers.

[0160] The initial machine learning model is trained using feature vectors to obtain a pre-trained machine learning model.

[0161] In one possible implementation, the second processing module 304 is specifically used for:

[0162] In response to detected anomalies, the end-to-end execution link view corresponding to the anomaly event is obtained based on a globally unique identifier;

[0163] The time consumption, error information and resource usage information of each level are compared and analyzed according to the execution sequence of the execution links in the end-to-end execution link view.

[0164] Identify the level that contributes the most to the end-to-end time consumption and / or the level that first produces an error as candidate failure points;

[0165] Within the hierarchy corresponding to the candidate fault point, log details, trace details, and storage access details under the same globally unique identifier are associated to output the fault location result. The fault location result includes at least the fault level, fault stage, fault type, and information corresponding to the fault.

[0166] In one possible implementation, the first generation module 301 is specifically used for:

[0167] Obtain the request initiation information and query statement information corresponding to the data query request. The request initiation information includes at least one of the following: initiation time, initiating entity identifier, and session identifier.

[0168] A globally unique identifier is generated based on the request initiation information and query statement information, and a mapping relationship is established between the globally unique identifier and the data query request. The mapping relationship is used to ensure consistency identification of the same data query request in the subsequent monitoring data collection, association and tracing process.

[0169] The data end-to-end monitoring device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0170] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0171] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0172] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0173] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0174] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0176] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0177] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and inject information into the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0179] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data end-to-end monitoring method, characterized in that, include: In response to a data query request, a globally unique identifier is generated based on the data query request; The globally unique identifier is injected and passed along the execution chain of the data analysis request; Based on the globally unique identifier, data association is performed on monitoring data from different levels to generate end-to-end execution status data reflecting the data query request; Anomaly detection and fault location are performed on the entire execution status data.

2. The method according to claim 1, characterized in that, The execution chain passes through at least the service portal layer, the business intelligence system layer, the interactive analysis engine computing layer, and the distributed file storage layer in sequence. The step of injecting and passing the globally unique identifier along the execution chain of the data analysis request includes: When the service portal layer receives the data query request, it injects the globally unique identifier into the request metadata; When the business intelligence system layer initiates a computation call to the interactive analysis engine computation layer, the globally unique identifier is injected into the call context or session attribute; When the interactive analysis engine's computation layer initiates data read / write access to the distributed file storage layer, the globally unique identifier is injected into the storage access context.

3. The method according to claim 1, characterized in that, The monitoring data at different levels includes: Request access logs and response time metrics for the service portal layer; Report parsing logs, query orchestration logs, and error code indicators at the business intelligence system layer; The interactive analysis engine's computational layer includes task scheduling logs, operator execution logs, resource usage metrics, and query phase time metrics. The distributed file storage layer includes read / write access logs, data block location information, and read / write latency metrics.

4. The method according to claim 3, characterized in that, The step of associating monitoring data from different levels based on the globally unique identifier to generate end-to-end execution status data reflecting the data query request includes: The collected monitoring data is cleaned, structured, parsed, and standardized to obtain standardized event data. Based on the globally unique identifier, the standardized event data is aggregated across layers and aligned in time sequence. Based on the aggregated data, the full-link execution status data is generated, which includes processing stages, time consumption, and status information at each level.

5. The method according to claim 1, characterized in that, The anomaly detection of the end-to-end execution status data includes: Extract time series of performance metrics associated with at least one of the globally unique identifiers from the end-to-end execution status data; The time series of the performance metrics are analyzed using a pre-trained machine learning model to identify abnormal patterns. The pre-trained machine learning model is trained in the following way: Acquire end-to-end execution status data within a historical time period as training samples; The training samples are subjected to feature engineering to extract feature vectors including time series statistical features, query text features, and link association features based on the globally unique identifier. The initial machine learning model is trained using the feature vectors to obtain the pre-trained machine learning model.

6. The method according to claim 1 or 4, characterized in that, The fault location process for the end-to-end execution status data includes: In response to detected anomaly information, an end-to-end execution link view corresponding to the anomaly event is obtained based on the globally unique identifier; The stage time, error information, and resource consumption information of each level are compared and analyzed according to the execution sequence of the execution links in the end-to-end execution link view. Identify the level that contributes the most to the end-to-end time consumption and / or the level that first produces an error as candidate failure points; In the hierarchy corresponding to the candidate fault point, log details, tracking details and storage access details under the same global unique identifier are associated, and fault location results are output. The fault location results include at least the fault level, fault stage, fault type and information corresponding to the fault.

7. The method according to claim 1, characterized in that, The step of generating a globally unique identifier in response to a data query request includes: Obtain the request initiation information and query statement information corresponding to the data query request. The request initiation information includes at least one of the following: initiation time, initiating entity identifier, and session identifier. A globally unique identifier is generated based on the request initiation information and the query statement information, and a mapping relationship is established between the globally unique identifier and the data query request. The mapping relationship is used to identify the consistency of the same data query request in the subsequent monitoring data collection, association and tracing process.

8. A data end-to-end monitoring device, characterized in that, include: The first generation module is used to generate a globally unique identifier in response to a data query request. The first processing module is used to inject and transmit the globally unique identifier along the execution link of the data analysis request; The second generation module is used to perform data association on monitoring data from different levels based on the globally unique identifier, and generate end-to-end execution status data reflecting the data query request. The second processing module is used to perform anomaly detection and fault location on the full-link execution status data.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images