A management system for enterprise-level observability analysis and a method of operation thereof

Abstract

This invention discloses an enterprise-level observability analysis management system and its operation method. The system includes: an observability data access layer for accessing observable data sources; a data processing layer for providing multiple data processing engines and using multiple data processing engines to process the observable data sources to obtain processing results; and a unified visualization layer for human-computer interaction and visualization of processing results. This invention can solve problems in existing observability practices such as signal silos, difficulties in cross-signal correlation, low root cause localization efficiency, lack of SLO management, and high costs associated with multiple systems.

Description

Technical Field

[0001] This invention relates to the field of computer software and big data processing technology, specifically to an enterprise-level observability analysis management system and its operation method. Background Technology

[0002] With the widespread adoption of microservice architectures and cloud-native technologies, the distributed complexity of enterprise application systems is growing exponentially. A single user request may traverse dozens of microservice nodes, involving hundreds of inter-service calls. In this context, observability has become a core capability for ensuring the reliable operation of distributed systems. Observability typically consists of three pillars: logs that record discrete events, metrics that reflect the time-series quantitative data of system state, and traces that record the complete call path of a request within the distributed system.

[0003] The following are the main problems with current observability practices in the industry:

[0004] (1) Signal silo problem: Logs, metrics and link traces are usually managed by different tools, such as ELK for logs, Prometheus+Grafana for metrics, and Jaeger for link traces. Three independent systems lead to data fragmentation. When troubleshooting, maintenance personnel need to switch frequently between multiple systems and manually associate data of different signals, which is time-consuming and easy to miss key information.

[0005] (2) Difficulty in correlation analysis: Existing tools lack built-in automatic correlation capabilities across signals. When a service is abnormal, the operation and maintenance personnel need to manually search for relevant logs in the log system, view the corresponding curves in the indicator system, and find relevant traces in the link system based on dimensions such as timestamp and service name. The whole process is highly dependent on human experience, and the mean time to recovery (MTTR) remains high.

[0006] (3) Low efficiency in root cause localization: In complex microservice topologies, a root cause failure may lead to cascading alarms from multiple downstream services, forming an alarm storm. Existing systems lack the ability to perform root cause analysis based on service dependency topologies, making it difficult for operations and maintenance personnel to quickly distinguish between root causes and symptoms.

[0007] (4) Lack of SLO management: Although most enterprises have defined service level objectives (SLO), they lack automated SLO monitoring and error budget tracking tools, and cannot make release decisions and capacity planning scientifically based on SLO data. (5) Storage and cost issues: The duplication of storage and computing resources in three independent systems consumes huge amounts of resources, and the total cost of ownership (TCO) is much higher than that of a unified platform solution. Summary of the Invention

[0008] To address the problems of signal silos, difficulties in cross-signal correlation, low root cause localization efficiency, lack of SLO management, and high costs of multiple systems in existing observability practices, the purpose of this invention is to provide an enterprise-level observability analysis management system and its operation method.

[0009] To achieve the above objectives, in a first aspect, embodiments of the present invention provide an enterprise-level observability analysis management system, comprising:

[0010] An observable data access layer is used to access observable data sources; these observable data sources include log data, metric data, trace data, front-end performance data, infrastructure, and custom events.

[0011] The data processing layer provides multiple data processing engines and uses multiple data sorting engines to process the observable data sources to obtain processing results.

[0012] The unified visualization layer provides a unified observable interface for SRE and DevOps engineers, and uses this unified observable interface to visualize human-computer interaction and processing results.

[0013] As one specific implementation of this application, the observability data access layer is specifically used for:

[0014] Supports the acquisition of observable data sources using multiple observable data protocols;

[0015] For semi-structured log data, automatically infer field types and dynamically generate summaries;

[0016] Automatically extract or inject associated markers; the associated markers include TraceID, SpanID, service name, and instance identifier.

[0017] As one specific implementation of this application, the data processing layer includes:

[0018] The streaming processing and correlation engine is used to perform cross-signal correlation tagging on each piece of data in the observable data source, to perform real-time parsing of the link tracing data to construct Span parent-child relationships, and to perform real-time pre-aggregation of high-frequency indicator data.

[0019] The log storage sub-engine is used to store the log data in Apache Parquet columnar format and supports full-text search in conjunction with a lightweight inverted index;

[0020] The time-series metric storage sub-engine is used to partition the metric data by time blocks. Each time block is organized into an independent time series by metric name and tag set, and stored in Parquet format before being written to object storage.

[0021] The link tracing storage sub-engine is used to build a hash partition index with TraceID as the primary key, and supports fast retrieval by TraceID complexity. Span data is stored in Parquet format and partitioned by time.

[0022] The cross-signal correlation analysis engine is used to automatically correlate and analyze log data, indicator data, and link tracing data using time window alignment algorithm, TraceID precise correlation algorithm, service topology automatic discovery algorithm, multidimensional root cause correlation sorting algorithm, and breadth-first search algorithm.

[0023] The SLO monitoring and error budget engine enables complete service level target lifecycle management, including SLI calculation, error budget tracking, and burn rate alerts.

[0024] The time window alignment algorithm is used to align the log data and the indicator data link tracing data by time.

[0025] The TraceID precise association algorithm is as follows: for log data and link tracing data carrying TraceID, a one-to-one association is achieved through precise matching of TraceID;

[0026] The service topology auto-discovery algorithm is as follows: automatically constructing a service topology graph based on the link tracing data; the service topology graph adopts a sliding window incremental update strategy.

[0027] The multidimensional root cause association ranking algorithm is as follows: when the system detects an anomaly, the root cause analysis engine performs root cause probability ranking on all related services based on the service topology graph.

[0028] The breadth-first search algorithm is as follows: trace upstream from the alarm node along the reverse direction of the topological edges of the service topology graph until the upstream anomaly source is found.

[0029] As a preferred implementation of this application, the data processing layer further includes an adaptive multi-level downsampling engine, used for:

[0030] Periodically perform downsampling aggregation on the expired data, converting high-precision aggregated data into low-precision aggregated data and writing it to the corresponding level.

[0031] As one specific implementation of this application, the unified observability interface includes:

[0032] Service map is a service topology graph based on automatic discovery, which displays the dependencies between services in an interactive directed graph format.

[0033] The request panoramic view is built based on user requests. When a user selects a specific TraceID, it can synchronously display the call chain waterfall chart of the user request, the log entries associated with each service node, and the corresponding indicator curves on the same interface, so as to realize one-click linkage viewing of the three signals.

[0034] The SLO dashboard displays the SLI achievement rate, error budget remaining, burn rate trend, and budget exhaustion prediction time for each service in real time.

[0035] Alarm context cards are used to display root cause analysis results, related log summaries, relevant metric snapshots, and topology impact range.

[0036] Secondly, embodiments of the present invention also provide an operating method for an enterprise-level observability analysis management system, applicable to the system described in the first aspect above, comprising:

[0037] Access observable data sources; these observable data sources include log data, metric data, tracing data, front-end performance data, infrastructure data, and custom events.

[0038] The system employs a streaming processing and correlation engine to perform cross-signal correlation tagging on each data point in the observable data source, performs real-time parsing of the link tracing data to construct Span parent-child relationships, and performs real-time pre-aggregation of high-frequency indicator data.

[0039] A cross-signal correlation analysis engine is used to automatically correlate and link log data, indicator data, and link tracing data, as well as perform anomaly detection and root cause localization analysis.

[0040] The SLO monitoring and error budget engine is used to complete the full service level target lifecycle management, including SLI calculation, error budget tracking, and burn rate alerts.

[0041] As a specific implementation of this application, the cross-signal association tag is as follows: a unified association context is attached to each data record, and a mapping index is established between log data, indicator data, and link tracing data; the association methods include precise association and fuzzy association.

[0042] The specific steps for constructing the parent-child relationship of a Span are as follows: the parentSpanID field in the link tracing data is parsed in real time, a call chain tree structure is constructed, and the service call relationship edges are extracted based on the service.name attribute of the Span.

[0043] As a preferred implementation of this application, the running method further includes:

[0044] Periodically perform downsampling aggregation on the expired data, converting high-precision aggregated data into low-precision aggregated data and writing it to the corresponding level.

[0045] As a preferred implementation of this application, the running method further includes:

[0046] It provides a unified observability interface for SRE and DevOps engineers, and uses the unified observability interface for human-computer interaction and visualization of processing results.

[0047] The unified observability interface includes:

[0048] Service map is a service topology graph based on automatic discovery, which displays the dependencies between services in an interactive directed graph format.

[0049] The request panoramic view is built based on user requests. When a user selects a specific TraceID, it can synchronously display the call chain waterfall chart of the user request, the log entries associated with each service node, and the corresponding indicator curves on the same interface, so as to realize one-click linkage viewing of the three signals.

[0050] The SLO dashboard displays the SLI achievement rate, error budget remaining, burn rate trend, and budget exhaustion prediction time for each service in real time.

[0051] Alarm context cards are used to display root cause analysis results, related log summaries, relevant metric snapshots, and topology impact range.

[0052] The advantages of implementing the embodiments of the present invention are as follows:

[0053] 1. Eliminate signal silos: Unify the collection of logs, metrics and link tracing to the same platform, and realize automatic correlation and linkage analysis of the three major signals through the built-in cross-signal correlation engine. Operation and maintenance personnel do not need to switch between multiple tools, which greatly improves the efficiency of fault diagnosis.

[0054] 2. Intelligent Root Cause Analysis: Based on automatic service topology discovery and multi-dimensional root cause association ranking algorithm, the root cause service can be accurately located from dozens of alarm services within seconds. Experimental results show that the root cause analysis accuracy rate reaches more than 85%, and the mean time to repair (MTTR) is shortened by about 60%.

[0055] 3. SLO-driven scientific operations: The complete SLO monitoring and error budget engine supports data-driven release decisions—allowing rapid iteration when the error budget is sufficient and automatically suggesting freezing changes when the budget is tight, realizing the transformation from passive alerts to proactive risk management.

[0056] 4. Significantly reduced infrastructure costs: A unified platform replaces three independent systems. Combined with a storage-compute separation architecture and multi-level downsampling strategies, the total infrastructure cost is reduced by approximately 70%. Adaptive downsampling reduces the data storage volume of 1-year indicators to only about 2% of the full-precision solution, greatly saving storage overhead without affecting long-term trend analysis.

[0057] 5. Real-time visibility of service topology: The automatic discovery algorithm can build and update the dependency graph between microservices in real time without manual configuration, and visualize the health status of services, traffic distribution and anomaly propagation paths, providing data support for architecture governance and capacity planning.

[0058] 6. High performance and low latency: The Rust-based computing engine, combined with Apache Arrow's vectorized execution, enables full-text log retrieval of TB-level data with an average response time of less than 2 seconds. Metric queries support second-level aggregation of millions of time series, and link tracing supports millisecond-level retrieval by TraceID.

[0059] 7. Open standards compatibility: Fully supports open source standards such as OpenTelemetry, Prometheus, Jaeger, and Zipkin, enabling enterprises to seamlessly migrate existing observability infrastructure and avoid vendor lock-in.

[0060] 8. Request-level panoramic insight: Through precise association driven by TraceID, it enables a panoramic view of the call chain, related logs and indicator status from the perspective of a single request, completely eliminating the problem of information fragmentation and enabling troubleshooting to return to a natural thinking mode centered on requests. Attached Figure Description

[0061] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0062] Figure 1 This is a management system architecture diagram for enterprise-level observability analysis provided in an embodiment of the present invention;

[0063] Figure 2 This is a flowchart of the cross-signal correlation analysis engine;

[0064] Figure 3 This is a flowchart of the automatic discovery and root cause analysis of service topology;

[0065] Figure 4 This is a flowchart of the SLO monitoring and error budget combustion engine workflow;

[0066] Figure 5 This is a flowchart of the time-series metric storage engine and adaptive downsampling process;

[0067] Figure 6 This is a flowchart of the operation method of the enterprise-level observability analysis management system provided in the embodiments of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0070] The inventive concept of this invention is to provide a high-performance enterprise-level observability platform system and its operating method. This system unifies the collection of three major observable signals—logs, metrics, and link tracing—onto a single platform, and achieves automatic correlation and linkage analysis of these three signals through an innovative cross-signal correlation analysis engine. This invention designs an automatic service topology discovery algorithm and a multi-dimensional root cause correlation ranking algorithm based on distributed link data, which can automatically construct service dependency graphs and locate the root cause service of a failure within seconds. Simultaneously, this invention implements a complete SLO monitoring and error budget burn rate engine, supporting hierarchical alarm strategies based on multi-window burn rate. This system is built using the Rust language, a storage-compute separation architecture, and unified columnar storage, which reduces infrastructure costs by approximately 70% compared to three independent system deployment schemes and significantly shortens the mean time to recovery.

[0071] Please refer to Figure 1 This is an enterprise-level observability analysis management system provided in this embodiment of the invention, comprising:

[0072] The observable data access layer (i.e., the unified data ingestion gateway in the diagram) is used to access observable data sources; the observable data sources include log data, indicator data, link tracing data, front-end performance data, infrastructure, and custom events;

[0073] The data processing layer provides multiple data processing engines and uses multiple data sorting engines to process the observable data sources to obtain processing results.

[0074] The unified visualization layer provides a unified observable interface for SRE and DevOps engineers, and uses this unified observable interface to visualize human-computer interaction and processing results.

[0075] It should be noted that, Figure 1 The main exhibit showcases the complete architecture and data flow relationships of the unified data ingestion gateway, streaming processing and correlation engine, three-signal storage engine, cross-signal correlation analysis engine, intelligent analysis and enterprise management layer, and unified visualization layer.

[0076] The following sections will introduce each part of the management system separately.

[0077] I. Observable Data Access Layer (Unified Data Ingestion Gateway)

[0078] This embodiment constructs a unified observability data access layer that supports multi-protocol adaptation. This layer is an innovative entry point that distinguishes this invention from traditional discrete solutions, and it can achieve the following functions:

[0079] (1) Multi-protocol adaptive access: The unified gateway supports multiple observable data protocols such as OpenTelemetry (OTLP gRPC / HTTP), Prometheus Remote Write, Jaeger Thrift / gRPC, Zipkin JSON, Syslog RFC 5424, and Fluentd Forward. The gateway automatically identifies the protocol type based on the request header and data format, without requiring explicit configuration by the user;

[0080] (2) Dynamic inference of the schema: For semi-structured log data, the system automatically infers the field type and dynamically generates the schema, supporting schema evolution. When the log format changes, there is no need to manually modify the configuration.

[0081] (3) Unified injection of association tags: During the data ingestion phase, association tags such as TraceID, SpanID, service name (service.name), and instance identifier (service.instance.id) are automatically extracted or injected to ensure the data foundation for subsequent cross-signal association analysis. For log data lacking TraceID, a fuzzy association strategy based on time window and service name is supported.

[0082] II. Data Processing Layer

[0083] The data processing layer in this embodiment includes Figure 1 The various engines in the system are as follows:

[0084] 1. Streaming and Correlation Engine

[0085] This engine is a high-performance streaming pipeline based on the Rust asynchronous runtime, performing real-time processing on received multi-signal data:

[0086] (1) Cross-signal correlation tagging: Attach a unified correlation context to each data record to establish a mapping index between logs, metrics, and link data. Correlation methods include exact correlation (accurate matching based on TraceID) and fuzzy correlation (approximate matching based on time window + service name + resource tag);

[0087] (2) Span parent-child relationship construction: The parentSpanID field in the link tracing data is parsed in real time to construct the call chain tree structure. The service call relationship edges are extracted based on the service.name attribute of the Span, laying the foundation for the construction of the service topology graph;

[0088] (3) Real-time pre-aggregation of indicators: Sliding window pre-aggregation is performed on high-frequency indicator data, and aggregated values ​​such as min, max, sum, count, and avg are calculated within a 60-second window granularity, reducing the backend storage and query pressure. The pre-aggregation results are written in parallel with the original data, and the query precision can be switched as needed.

[0089] 2. Multi-signal unified storage engine

[0090] This engine features separate storage sub-engines optimized for three data characteristics: logs, metrics, and trace links, while sharing a common underlying architecture of storage-compute separation.

[0091] 1. Log storage sub-engine: The log data is stored in Apache Parquet columnar format and combined with a lightweight inverted index to support full-text search.

[0092] Among them, the lightweight inverted index: a lightweight inverted index is built for each Parquet file during the data ingestion stage to record the mapping relationship from keywords to the file level, which is used to quickly locate the data file containing the target keywords. The index size is only 1% to 3% of the original data.

[0093] (2) Time-series index storage sub-engine: This sub-engine is specifically optimized for the time-series characteristics of index data. Delta-of-Delta encoding is used to process the timestamp sequence, minimizing encoding overhead by leveraging the equal intervals of the timestamps. The encoding formula is:

[0094]

[0095] In scenarios where data is collected at equal intervals (e.g., once every 15 seconds). Approaching zero, a timestamp requires only 1 to 2 bits after Zigzag variable-length encoding. The index value uses XOR encoding, taking advantage of the characteristic that the values ​​of adjacent sampling points usually change little, encoding only the changing bits to achieve an extremely high compression ratio.

[0096] The indicator data is partitioned into 2-hour blocks. Within each block, the data is organized into independent time series based on indicator name and tag set. The data is stored in Parquet format and then written to object storage.

[0097] (3) Link tracing storage sub-engine: A hash partition index is built with TraceID as the primary key, which supports fast retrieval by TraceID with O(1) complexity. Span data is stored in Parquet format by time partition, and an inverted index is maintained to support queries by service.name, operation.name, span.status and other dimensions.

[0098] 3. Cross-signal correlation analysis engine

[0099] The engine's workflow is as follows: Figure 2 As shown, it mainly demonstrates a five-step association analysis process, from context injection, time window alignment, TraceID precise association, service topology construction to root cause ranking.

[0100] This engine enables automatic correlation and linkage analysis of three major observable signals. Specifically, it includes the following key algorithms:

[0101] (1) Time window alignment algorithm: Events of different signals are aligned according to a unified time window to ensure temporal consistency in correlation analysis. For event e, the formula for calculating the aligned time window is:

[0102]

[0103] Where t(e) is the original timestamp of the event, and Δt is the alignment granularity (default 60 seconds). This is a function for rounding down. After alignment, log, metric, and Span data within the same window can be linked for queries;

[0104] (2) TraceID Precise Association: For log and link data carrying TraceID, one-to-one association is achieved through precise TraceID matching. The system automatically constructs a panoramic view at the request level, displaying the complete call chain of the request, the log output of each node, and the service indicator status at the corresponding time on the same interface;

[0105] (3) Service Topology Auto-Discovery Algorithm: Based on real-time link tracing data, a directed graph G(V, E) of service dependencies is automatically constructed, where:

[0106] V = {s | s ∈ services}

[0107] E={(s_caller,s_callee)|∃span: span.service = s_callee∧span.parent.service = s_caller}

[0108] Each edge E includes aggregated statistical attributes: call count, error count, latency percentile (latency_p50, latency_p99), and protocol type (HTTP / gRPC / TCP). The topology graph uses a sliding window incremental update strategy, with a default window width W of 5 minutes and a step size S of 1 minute to avoid the computational overhead of full reconstruction.

[0109] G_t = Merge(G_{t-1}, ΔG_t)

[0110] Where G_t is the topological snapshot at time t, and ΔG_t is the increment of newly observed call relationships within the most recent step. The Merge operation updates the statistical attributes of existing edges and inserts new edges;

[0111] (4) Multidimensional Root Cause Association Ranking Algorithm: When the system detects an anomaly, the root cause analysis engine ranks all related services by root cause probability based on the service topology graph. The root cause score calculation formula for each service node s is as follows:

[0112] Score(s) = w1×ΔError(s) + w2×ΔLatency(s) + w3×Impact(s) + w4×Anomaly(s)

[0113] Wherein, ΔError(s) is the change in error rate of service s relative to the baseline within the abnormal time window (normalized to the 0-1 range), ΔLatency(s) is the change in latency percentile relative to the baseline, Impact(s) is the downstream impact range of the service (the proportion of traffic to affected downstream services to the total traffic), and Anomaly(s) is the anomaly significance determined based on a statistical detection algorithm. w1 to w4 are weighting coefficients, with default values ​​of 0.3, 0.25, 0.25, and 0.2 respectively, which can be customized according to business scenarios.

[0114] It should be noted that the service topology automatic discovery and root cause localization process is as follows: Figure 3 As shown, it mainly demonstrates the complete process of automatically extracting service call relationships from Span data, constructing a topology map, analyzing anomaly propagation paths, assessing the scope of impact, and outputting root cause ranking.

[0115] 5. Breadth-First Search (BFS) Algorithm

[0116] The anomaly propagation path tracing uses the BFS algorithm, tracing upstream from the alarm node along the reverse direction of the topological edges until the upstream anomaly source is found:

[0117] Path = BFS_reverse(G, alert_node, depth_limit)

[0118] The depth_limit parameter specifies the maximum tracing depth (default 5 layers) to prevent over-searching in large-scale topologies.

[0119] 4. SLO monitoring and error budgeting engine

[0120] The engine's workflow is as follows: Figure 4 As shown, it mainly demonstrates the complete SLO monitoring pipeline from SLO definition input, SLI real-time calculation, error budget calculation, combustion rate calculation to multi-window hierarchical alarm.

[0121] This engine enables complete Service Level Objective (SLO) lifecycle management, including SLI calculation, error budget tracking, and burn rate alerts.

[0122] (1) Real-time calculation of SLI (Service Level Indicator): Supports multiple SLI types based on request success rate, latency threshold, throughput, etc. Taking availability SLI as an example, the calculation formula is:

[0123] SLI_availability = (good_events / total_events) × 100%

[0124] Where good_events is the number of requests that meet the SLO standard (e.g., HTTP status code <500), and total_events is the total number of requests. SLI is continuously calculated using a sliding window method, supporting evaluation windows from 1 minute to 30 days;

[0125] (2) Error Budget Calculation and Tracking: The error budget represents the maximum unavailability allowed within the SLO commitment. The formulas for calculating the total budget and remaining budget are as follows:

[0126] EB_total = (1 - SLO_target) × window_duration

[0127] EB_remaining = EB_total - Σ(bad_minutes)

[0128] EB_consumption = 1 - EB_remaining / EB_total

[0129] For example, with a 99.9% SLO target and a 30-day evaluation window, the total error budget is 43.2 minutes. The system tracks budget consumption in real time and predicts budget exhaustion time based on the consumption rate.

[0130] (3) Multi-window Burn Rate Alarm: This invention adopts the multi-window burn rate alarm strategy from Google SRE practices. The burn rate is defined as the ratio of the actual error rate to the allowable error rate:

[0131] BurnRate = actual_error_rate / (1 - SLO_target)

[0132] BurnRate = 1 indicates that the entire budget is exhausted exactly at the end of the evaluation window, while BurnRate > 1 indicates that the consumption rate exceeds expectations. The system has two alarm window settings: a short window (1 hour) detects rapid burning (triggers an emergency alarm if BurnRate > 14.4), and a long window (6 hours) detects slow burning (triggers a warning alarm if BurnRate > 6). The dual-window strategy balances alarm sensitivity and false alarm rate.

[0133] 5. Adaptive multi-level downsampling engine

[0134] To optimize performance for long-range queries, this invention designs an automated multi-level downsampling mechanism:

[0135] (1) Definition of downsampling levels: The system maintains four precision levels: the original precision level (retains the complete sampling interval, with a default retention period of 2 days), the 5-minute aggregation level (retains min / max / avg / count / sum, with a default retention period of 30 days), the 1-hour aggregation level (retains the statistical summary, with a default retention period of 1 year), and the 1-day aggregation level (retains the long-term trend, with no limit on the retention period).

[0136] (2) Automatic downsampling execution: The background Compactor thread periodically performs downsampling aggregation on expired data, converting high-precision data into low-precision aggregated data and writing it to the corresponding level. The storage savings brought by downsampling are calculated using the following formula:

[0137] S_saved = S_raw × (1 - 1 / R)

[0138] Where S_raw is the original data storage size, and R is the downsampling ratio (e.g., R = 20 if reducing from 15 seconds to 5 minutes). After four levels of downsampling, the storage size of the 1-year index data is only about 2% of that of the full-precision retention scheme;

[0139] (3) Automatic Query Routing: The PromQL-compatible query engine automatically selects the optimal precision level based on the query's time range. The routing rules are as follows: the original precision is used when the query range is less than 2 days, the 5-minute aggregation level is used when the query range is 2 to 30 days, the 1-hour aggregation level is used when the query range is 30 days to 1 year, and the 1-day aggregation level is used when the query range exceeds 1 year. Automatic routing is completely transparent to the user and requires no modification to the query statement.

[0140] It should be noted that the time-series metric storage engine and the adaptive downsampling process are as follows: Figure 5 As shown, it mainly displays the complete storage and query path of indicator data, from real-time pre-aggregation, time-series dedicated encoding, time-partitioned storage to four-level adaptive downsampling.

[0141] III. Unified Visualization and Interaction Layer

[0142] This layer provides a unified observability interface for SRE and DevOps engineers:

[0143] (1) Service Map: Based on the automatically discovered service topology graph, it displays the dependencies between services in an interactive directed graph. The color coding of nodes reflects the health status of services (normal / warning / abnormal), the thickness of the edges reflects the traffic volume, and the mouse hover displays real-time request rate, error rate and latency indicators;

[0144] (2) Request panoramic view: When the user selects a specific TraceID, the system synchronously displays the call chain waterfall chart of the request, the log entries associated with each service node and the corresponding indicator curves on the same interface, realizing one-click linkage viewing of the three signals;

[0145] (3) SLO Dashboard: Real-time display of SLI achievement rate, remaining error budget, burn rate trend, and budget exhaustion prediction time for each service. Supports multi-level drill-down analysis by service, environment, and time dimensions;

[0146] (4) Alarm Context Card: When an alarm is triggered, the system automatically generates a rich context alarm card containing root cause analysis results, related log summaries, relevant indicator snapshots and the topology of the affected area, to help on-duty personnel quickly understand the complete context of the alarm.

[0147] As can be seen from the above description, the advantages of implementing the embodiments of the present invention are as follows:

[0148] 1. Eliminate signal silos: Unify the collection of logs, metrics and link tracing to the same platform, and realize automatic correlation and linkage analysis of the three major signals through the built-in cross-signal correlation engine. Operation and maintenance personnel do not need to switch between multiple tools, which greatly improves the efficiency of fault diagnosis.

[0149] 2. Intelligent Root Cause Analysis: Based on automatic service topology discovery and multi-dimensional root cause association ranking algorithm, the root cause service can be accurately located from dozens of alarm services within seconds. Experimental results show that the root cause analysis accuracy rate reaches more than 85%, and the mean time to repair (MTTR) is shortened by about 60%.

[0150] 3. SLO-driven scientific operations: The complete SLO monitoring and error budget engine supports data-driven release decisions—allowing rapid iteration when the error budget is sufficient and automatically suggesting freezing changes when the budget is tight, realizing the transformation from passive alerts to proactive risk management.

[0151] 4. Significantly reduced infrastructure costs: A unified platform replaces three independent systems. Combined with a storage-compute separation architecture and multi-level downsampling strategies, the total infrastructure cost is reduced by approximately 70%. Adaptive downsampling reduces the data storage volume of 1-year indicators to only about 2% of the full-precision solution, greatly saving storage overhead without affecting long-term trend analysis.

[0152] 5. Real-time visibility of service topology: The automatic discovery algorithm can build and update the dependency graph between microservices in real time without manual configuration, and visualize the health status of services, traffic distribution and anomaly propagation paths, providing data support for architecture governance and capacity planning.

[0153] 6. High performance and low latency: The Rust-based computing engine, combined with Apache Arrow's vectorized execution, enables full-text log retrieval of TB-level data with an average response time of less than 2 seconds. Metric queries support second-level aggregation of millions of time series, and link tracing supports millisecond-level retrieval by TraceID.

[0154] 7. Open standards compatibility: Fully supports open source standards such as OpenTelemetry, Prometheus, Jaeger, and Zipkin, enabling enterprises to seamlessly migrate existing observability infrastructure and avoid vendor lock-in.

[0155] 8. Request-level panoramic insight: Through precise association driven by TraceID, it enables a panoramic view of the call chain, related logs and indicator status from the perspective of a single request, completely eliminating the problem of information fragmentation and enabling troubleshooting to return to a natural thinking mode centered on requests.

[0156] Based on the same inventive concept, this invention provides an operating method applicable to the enterprise-level observability analysis management system described above. For example... Figure 6 As shown, the running method includes:

[0157] S1, platform deployment and multi-signal access.

[0158] The management system in the foregoing embodiments can also be understood as a platform. Before running this platform, the following deployment is required:

[0159] (1) Deploy the observability platform components in the Kubernetes cluster. Deploy OpenTelemetry Collector as a unified data collection agent, configure Receiver to receive data from multiple protocols such as OTLP, Prometheus, and Jaeger, and forward logs, metrics, and tracing data to the platform's ingestion gateway through Pipeline.

[0160] (2) Deploy the OpenTelemetry SDK (which supports multiple languages ​​such as Java, Go, Python, and Node.js) on the application side to achieve application-level automatic instrumentation. The SDK automatically generates a TraceID for each inbound request and propagates context information through the W3C Trace Context standard during inter-service calls to ensure that logs, metrics, and trace data carry a unified association identifier.

[0161] (3) Configure Prometheus scraping rules (Scrape Config) to collect infrastructure metrics (CPU, memory, disk, network) and application-defined metrics, and push them to the platform's metric storage engine in a unified manner through the Remote Write protocol.

[0162] After deployment is complete, observable data sources can be accessed, including but not limited to log data, metric data, tracing data, front-end performance data, infrastructure data, and custom events.

[0163] S2, Real-time Stream Processing and Topology Building.

[0164] In specific implementation, a streaming processing and correlation engine is used to perform cross-signal correlation tagging on each data point in the observable data source, to perform real-time parsing of the link tracing data to construct Span parent-child relationships, and to perform real-time pre-aggregation of high-frequency indicator data, as detailed below:

[0165] (1) After the ingestion gateway receives multi-signal data, the streaming processing engine supplements the associated context for each data record. Taking an application log as an example: the engine extracts the TraceID field from the log text (supports regular expressions and JSON path extraction), and at the same time obtains service.name and instance.id from the metadata to build a complete set of associated keys {traceID, service, instance, timestamp}.

[0166] (2) The link tracing pipeline extracts the call relationship triples (caller_service, callee_service, attributes) from each Span and writes them into the topology relationship buffer. The topology aggregator aggregates the relationship triples within the last 5 minutes (window W) every 1 minute (step S), counts the number of requests, errors and latency distribution for each call edge, and updates the service topology graph G(V,E).

[0167] (3) The indicator pre-aggregation engine performs rolling aggregation on the received raw indicator data in a 60-second window. For Counter type indicators, the rate is calculated, and for Histogram type indicators, percentiles (p50 / p95 / p99) are calculated. The aggregated indicators are written to storage in parallel with the original indicators; the former is used for dashboard display, and the latter is used for precise queries.

[0168] S3, cross-signal correlation query.

[0169] In practice, cross-signal correlation analysis employs a time window alignment algorithm, a TraceID precise correlation algorithm, a service topology automatic discovery algorithm, a multidimensional root cause correlation sorting algorithm, and a breadth-first search algorithm to automatically correlate and analyze log data, indicator data, and link tracing data. For a more detailed processing flow, please refer to the aforementioned embodiments; it will not be repeated here.

[0170] More specifically, when a user discovers an abnormal error log in the log explorer, clicking on the TraceID field of that log record triggers an automatic cross-signal correlation query: first, it retrieves the complete call chain corresponding to the TraceID from the link storage; then, it retrieves log entries for all service nodes under the same TraceID from the log storage; and finally, it queries the metrics storage for key metrics (error rate, latency p99, CPU utilization, etc.) of the relevant services within 5 minutes before and after the anomaly occurred. All three sets of data are aligned along the timeline and displayed uniformly in the request panorama view.

[0171] When a user finds a service node highlighted in red (indicating an abnormal state) on the service map, clicking on the node will automatically expand the correlation analysis panel, displaying the service's real-time metric trends, recent error log summaries, upstream and downstream call relationships, and a list of the slowest traces, helping the user quickly define the scope of the problem.

[0172] S4, Anomaly Detection and Root Cause Analysis.

[0173] In practice, anomaly detection and root cause analysis mainly include:

[0174] (1) The multi-dimensional anomaly detection engine continuously monitors the key metrics (error rate, latency, traffic) of all services. When an anomaly event is triggered when a metric deviates from the dynamic baseline by more than a threshold, the anomaly detection uses the EWMA (Exponential Weighted Moving Average) algorithm, with a smoothing coefficient λ = 2 / (span+1), and the alarm threshold is the dynamic mean plus or minus k times the standard deviation (k defaults to 3).

[0175] (2) Upon triggering an abnormal event, the root cause analysis engine is immediately activated. The engine first extracts the upstream and downstream subgraphs of the abnormal node from the topology graph, and then performs a breadth-first search (BFS) in the reverse direction of the call chain, checking the health status of upstream services layer by layer. For each service node on the search path, a root cause score Score(s) is calculated, taking into account four dimensions: error rate variation, latency variation, downstream impact range, and anomaly significance. The final output is a ranking list of root cause services arranged in descending order of score.

[0176] (3) The system automatically generates a root cause analysis report, which includes: root cause service ranking (with confidence percentage), abnormal propagation path visualization (highlighted on the topology map), associated abnormal log entries (the top 10 error logs sorted by time), and impact scope assessment (a list of affected downstream services and an estimated user impact scope).

[0177] S5, SLO monitoring and budget management.

[0178] In practice, SLO monitoring and budget management mainly include:

[0179] (1) The operation and maintenance personnel configure the SLO target in the SLO management interface. Taking an API gateway service as an example: select "Availability" for SLI type, define the good_event condition as "HTTP status code <500", set the SLO target to 99.9%, and set the evaluation window to a 30-day rolling window.

[0180] (2) The system calculates the SLI achievement rate and the progress of erroneous budget consumption in real time. The SLO dashboard displays: current SLI value (e.g., 99.95%), total erroneous budget (43.2 minutes), consumed budget (12.5 minutes), remaining budget (30.7 minutes), current burn rate (0.85), and predicted time of budget exhaustion.

[0181] (3) The system automatically triggers graded alarms based on the burning rate. When the BurnRate in the 1-hour window exceeds 14.4 (equivalent to consuming 100% of the monthly budget in 1 hour), an emergency alarm is triggered and the on-duty personnel are notified. When the BurnRate in the 6-hour window exceeds 6, a warning-level alarm is triggered.

[0182] (4) When the error budget exceeds 80%, the system will automatically send a change freeze suggestion to the team and set an approval gate in the release pipeline, requiring the SLO manager to approve before the release can be executed.

[0183] S6, Indicator Downsampling and Data Lifecycle.

[0184] In this embodiment, the method periodically performs downsampling aggregation on the periodic data, converting high-precision aggregated data into low-precision aggregated data and writing it to the corresponding level. For a more detailed description, please refer to the aforementioned system embodiment section.

[0185] For example, step S6 includes:

[0186] (1) The background Compactor thread executes downsampling tasks periodically according to the configured retention strategy. Taking the default configuration as an example: after retaining the original precision data for 2 days, a 5-minute granularity downsampling aggregation is triggered, calculating the statistical values ​​such as min / max / avg / count / sum / p50 / p99 within each 5-minute window and writing them into the 5-minute aggregation layer; after 30 days, 1-hour downsampling is executed; after 1 year, 1-day downsampling is executed.

[0187] (2) When parsing a query request, the PromQL query engine automatically selects the optimal precision layer based on the query_range parameter. The selection logic is as follows: under the premise of satisfying the coverage of the query time range, the finest available layer is selected. For example, if a user queries the CPU usage trend of the last 7 days, the engine automatically routes to the 5-minute aggregation layer (covering the last 30 days), returning approximately 2,016 data points (7 days × 24 hours × 12 points / hour), which ensures the display accuracy and avoids the performance overhead of scanning the original data.

[0188] (3) The data lifecycle manager automatically cleans up expired data and releases object storage space according to the retention policy configured for each data stream. It supports configuring different retention periods by organization, data stream, and data type (log / metric / link).

[0189] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A management system for enterprise-level observability analysis, characterized in that, include: The observability data access layer is used to access observable data sources; The observable data sources include log data, metric data, tracing data, front-end performance data, infrastructure, and custom events; The data processing layer provides multiple data processing engines and uses multiple data sorting engines to process the observable data sources to obtain processing results. The unified visualization layer provides a unified observable interface for SRE and DevOps engineers, and uses this unified observable interface to visualize human-computer interaction and processing results.

2. The management system as described in claim 1, characterized in that, The observable data access layer is specifically used for: Supports the acquisition of observable data sources using multiple observable data protocols; For semi-structured log data, automatically infer field types and dynamically generate summaries; Automatically extract or inject associated markers; the associated markers include TraceID, SpanID, service name, and instance identifier.

3. The management system as described in claim 1, characterized in that, The data processing layer includes: The streaming processing and correlation engine is used to perform cross-signal correlation tagging on each piece of data in the observable data source, to perform real-time parsing of the link tracing data to construct Span parent-child relationships, and to perform real-time pre-aggregation of high-frequency indicator data. The log storage sub-engine is used to store the log data in Apache Parquet columnar format and supports full-text search in conjunction with a lightweight inverted index; The time-series metric storage sub-engine is used to partition the metric data by time blocks. Each time block is organized into an independent time series by metric name and tag set, and stored in Parquet format before being written to object storage. The link tracing storage sub-engine is used to build a hash partition index with TraceID as the primary key, and supports fast retrieval by TraceID complexity. Span data is stored in Parquet format and partitioned by time. The cross-signal correlation analysis engine is used to automatically correlate and analyze log data, indicator data, and link tracing data using time window alignment algorithm, TraceID precise correlation algorithm, service topology automatic discovery algorithm, multidimensional root cause correlation sorting algorithm, and breadth-first search algorithm. The SLO monitoring and error budget engine enables complete service level target lifecycle management, including SLI calculation, error budget tracking, and burn rate alerts.

4. The management system as described in claim 3, characterized in that, The time window alignment algorithm is used to align the log data and indicator data link tracing data by time. The TraceID precise association algorithm is as follows: for log data and link tracing data carrying TraceID, a one-to-one association is achieved through precise matching of TraceID; The service topology automatic discovery algorithm is as follows: automatically construct a service topology graph based on the link tracing data; The service topology map adopts a sliding window incremental update strategy; The multidimensional root cause association ranking algorithm is as follows: when the system detects an anomaly, the root cause analysis engine performs root cause probability ranking on all related services based on the service topology graph. The breadth-first search algorithm is as follows: trace upstream from the alarm node along the reverse direction of the topological edges of the service topology graph until the upstream anomaly source is found.

5. The management system as described in claim 3, characterized in that, The data processing layer also includes an adaptive multi-level downsampling engine for: Periodically perform downsampling aggregation on the expired data, converting high-precision aggregated data into low-precision aggregated data and writing it to the corresponding level.

6. The management system as described in claim 3, characterized in that, The unified observability interface includes: Service map is a service topology graph based on automatic discovery, which displays the dependencies between services in an interactive directed graph format. The request panoramic view is built based on user requests. When a user selects a specific TraceID, it can synchronously display the call chain waterfall chart of the user request, the log entries associated with each service node, and the corresponding indicator curves on the same interface, so as to realize one-click linkage viewing of the three signals. The SLO dashboard displays the SLI achievement rate, error budget remaining, burn rate trend, and budget exhaustion prediction time for each service in real time. Alarm context cards are used to display root cause analysis results, related log summaries, relevant metric snapshots, and topology impact range.

7. A method for operating an enterprise-level observability analysis management system, characterized in that, The operating method is applicable to the management system as described in any one of claims 1-6, including: Access observable data sources; these observable data sources include log data, metric data, tracing data, front-end performance data, infrastructure data, and custom events. The system employs a streaming processing and correlation engine to perform cross-signal correlation tagging on each data point in the observable data source, performs real-time parsing of the link tracing data to construct Span parent-child relationships, and performs real-time pre-aggregation of high-frequency indicator data. A cross-signal correlation analysis engine is used to automatically correlate and link log data, indicator data, and link tracing data, as well as perform anomaly detection and root cause localization analysis. The SLO monitoring and error budget engine is used to complete the full service level target lifecycle management, including SLI calculation, error budget tracking, and burn rate alerts.

8. The operating method as described in claim 7, characterized in that, Cross-signal association tagging specifically involves attaching a unified association context to each data record and establishing a mapping index between log data, indicator data, and link tracing data; the association methods include precise association and fuzzy association. The specific steps for constructing the parent-child relationship of a Span are as follows: the parentSpanID field in the link tracing data is parsed in real time, a call chain tree structure is constructed, and the service call relationship edges are extracted based on the service.name attribute of the Span.

9. The operating method as described in claim 7, characterized in that, The operating method further includes: Periodically perform downsampling aggregation on the expired data, converting high-precision aggregated data into low-precision aggregated data and writing it to the corresponding level.

10. The operating method as described in claim 9, characterized in that, The operating method further includes: It provides a unified observability interface for SRE and DevOps engineers, and uses the unified observability interface for human-computer interaction and visualization of processing results. The unified observability interface includes: Service map is a service topology graph based on automatic discovery, which displays the dependencies between services in an interactive directed graph format. The request panoramic view is built based on user requests. When a user selects a specific TraceID, it can synchronously display the call chain waterfall chart of the user request, the log entries associated with each service node, and the corresponding indicator curves on the same interface, so as to realize one-click linkage viewing of the three signals. The SLO dashboard displays the SLI achievement rate, error budget remaining, burn rate trend, and budget exhaustion prediction time for each service in real time. Alarm context cards are used to display root cause analysis results, related log summaries, relevant metric snapshots, and topology impact range.

Images