Ceph monitoring, log and link tracking integrated method and system

Abstract

The invention discloses a Ceph monitoring, log and link tracking integrated method and system, belongs to the technical field of distributed storage system monitoring, and aims to solve the technical problem of how to improve the overall observability and alarm capability of a Ceph cluster and the fault diagnosis capability of a distributed system. Monitoring indexes, log data and link tracking data are collected in a unified mode; the monitoring indexes, the log data and the link tracking data are sent to the OpenObserve in a unified mode; a multi-level alarm is generated in real time according to a log mode, an index threshold value and a link error rate through an alarm engine built in OpenObserve, and a consumption interface is provided for ceh-mgr through REST API; and the complete call chain of the client request in the Ceph MON component, the OSD component and the MDS component is visually displayed through the OpenObserve Tracing UI.

Description

Technical Field

[0001] This invention relates to the field of distributed storage system monitoring technology, specifically to an integrated method and system for Ceph monitoring, logging, and tracing. Background Technology

[0002] Traditional Ceph monitoring solutions primarily rely on Prometheus Server and Alertmanager, which face numerous technical bottlenecks and operational challenges in real-world production environments. Specific issues include:

[0003] 1. Limited functionality: Existing technologies can only collect and create alarms, but cannot analyze complex information such as latency and errors in distributed systems, and lack the ability to gain deep insights into system performance;

[0004] 2. Complex configuration: Prometheus Server and Alertmanager require two separate interfaces to manage alert rules and the silent mechanism, which leads to a sharp increase in configuration and operation and maintenance complexity, and increases the risk of human error.

[0005] 3. Strong dependence on visualization: Traditional solutions cannot provide built-in visualization functions and must rely on third-party tools such as Grafana to realize the visualization of monitoring information, which increases the complexity of system architecture and maintenance costs.

[0006] 4. Poor data collection timeliness: Prometheus Server uses a passive pull mode to obtain cluster status. When the cluster status changes frequently, it cannot obtain the instantaneous abnormal status of the cluster within the scanning cycle interval, resulting in the loss of critical abnormal information.

[0007] 5. Distributed data management: Monitoring metrics and log data are stored in different systems, which is not conducive to centralized management, correlation analysis, and troubleshooting.

[0008] 6. Lack of distributed tracing capabilities: Traditional solutions cannot provide distributed tracing functionality. When complex cross-component failures occur in the Ceph cluster, it is difficult to quickly locate the root cause of the problem, especially in scenarios where client requests involve multiple OSD, MDS, or MON components, lacking end-to-end request chain visualization capabilities.

[0009] 7. Low efficiency in fault location: The lack of a unified observability platform requires maintenance personnel to switch between multiple systems to find problems, which affects the efficiency of fault location and repair.

[0010] The aforementioned technical deficiencies severely impact the real-time performance, accuracy, completeness, and fault diagnosis capabilities of Ceph cluster monitoring, making it difficult to meet the demands of modern distributed storage systems for high-precision monitoring and rapid fault location.

[0011] Improving the overall observability, alarm capabilities, and distributed system fault diagnosis capabilities of Ceph clusters is a technical problem that needs to be solved. Summary of the Invention

[0012] The technical objective of this invention is to address the above-mentioned shortcomings by providing an integrated method and system for Ceph monitoring, logging, and tracing, thereby solving the technical problem of how to improve the overall observability, alarm capabilities, and distributed system fault diagnosis capabilities of Ceph clusters.

[0013] In a first aspect, the present invention provides an integrated method for Ceph monitoring, logging, and tracing, applied to a Ceph cluster. Each node in the Ceph cluster deploys OpenTelemetry Collector as a data acquisition layer, OpenTelemetry Protocol as a data transmission layer, OpenObserve as a data processing and display layer, and OpenObserve Tracing UI as a tracing visualization layer. The method includes the following steps:

[0014] Data Acquisition: Monitoring metrics, log data, and link tracing data are collected uniformly through the OpenTelemetry Collector;

[0015] Data transmission: Based on the OpenTelemetry protocol, monitoring metrics, log data, and tracing data are uniformly sent to OpenObserver via the OpenTelemetry Protocol;

[0016] Data processing and display: Within OpenObserve, monitoring metrics, log data, and trace data are written to the same database and a cascading index of trace_id and span_id is created. Through OpenObserve's built-in alarm engine, multi-level alarms are generated in real time based on log patterns, metric thresholds, and trace error rates, and a consumption interface is provided to ceph-mgr via a REST API.

[0017] Link tracing visualization: The OpenObserve Tracing UI visualizes the complete call chain of client requests in Ceph MON, OSD, and MDS components, and correlates it with the monitoring metrics, log data, and link tracing data corresponding to the same trace_id.

[0018] As a preferred method, OpenTelemetry Collector obtains cluster status data from Ceph Manager in real time and collects log data from Ceph OSD, Ceph MDS and Ceph MON processes in real time through file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it traces the complete execution path of client requests within the Ceph cluster to obtain link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD and MDS components.

[0019] As a preferred option, OpenTelemetry Collector sends data in a batch processing + compression + retry manner, and supports local caching and automatic replay when the network is interrupted.

[0020] As a preferred approach, OpenObserve employs a hybrid architecture combining columnar storage and inverted indexes.

[0021] Secondly, this invention provides an integrated system for Ceph monitoring, logging, and tracing, applied to a Ceph cluster, comprising OpenTelemetry Collector as the data acquisition layer, OpenTelemetryProtocol as the data transmission layer, OpenObserve as the data processing and display layer, and OpenObserveTracing UI as the tracing visualization layer.

[0022] OpenTelemetry Collector is used to uniformly collect monitoring metrics, log data, and link tracing data;

[0023] The OpenTelemetry Protocol is used to uniformly send monitoring metrics, log data, and link tracing data to OpenObserver based on the OpenTelemetry protocol;

[0024] OpenObserve is used to write monitoring metrics, log data, and trace data to the same database and create cascading indexes of trace_id and span_id. It is used to generate multi-level alarms in real time based on log patterns, metric thresholds, and trace error rates through the built-in alarm engine, and provides a consumption interface to ceph-mgr through a REST API.

[0025] The OpenObserve Tracing UI is used to visualize the complete call chain of client requests in Ceph MON, OSD, and MDS components, and to associate it with the monitoring metrics, log data, and tracing data corresponding to the same trace_id.

[0026] As a preferred option, OpenTelemetry Collector is used to obtain cluster status data from Ceph Manager in real time and collect log data from Ceph OSD, Ceph MDS and Ceph MON processes in real time through file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it tracks the complete execution path of client requests within the Ceph cluster to obtain link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD and MDS components.

[0027] As a preferred option, OpenTelemetry Collector is used to send data in a batch + compression + retry manner, and supports local caching and automatic replay when the network is interrupted.

[0028] As a preferred approach, OpenObserve employs a hybrid architecture combining columnar storage and inverted indexes.

[0029] The integrated method and system for Ceph monitoring, logging, and link tracing of the present invention have the following advantages:

[0030] 1. Improve monitoring efficiency: By uniformly collecting three types of telemetry data through OpenTelemetry Collector, the complexity of maintaining multiple systems is reduced, and operation and maintenance costs are reduced by more than 60%;

[0031] 2. Enhanced fault location capabilities: Based on OpenObserve's tracing function, it can quickly locate performance bottlenecks and fault points in distributed systems, improving fault location efficiency by more than 80%.

[0032] 3. Achieve end-to-end observability: Provide end-to-end visualization capabilities from the client to the storage cluster through unified telemetry data acquisition and correlation analysis;

[0033] 4. Simplified configuration management: A single OpenTelemetry Collector platform reduces the configuration management burden of multiple systems, lowering operational complexity and the risk of human error;

[0034] 5. Eliminate toolchain dependencies: The built-in OpenObserve visualization and tracing functions completely replace tools such as Grafana and Jaeger, reducing dependencies on third-party components and lowering the complexity of the system architecture;

[0035] 6. Improve data integrity: The real-time push mechanism ensures that abnormal cluster states can be captured in a timely manner, avoiding data loss problems caused by the traditional pull mode;

[0036] 7. Enhanced real-time response capabilities: The event-driven monitoring mode significantly improves the timeliness of anomaly detection and alarm response. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] The invention will be further described below with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart of an integrated Ceph monitoring, logging, and tracing method as described in Example 1. Figure 2 This is a schematic diagram illustrating the working principle of an integrated Ceph monitoring, logging, and link tracing system as described in Example 1. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0041] This invention provides an integrated method and system for Ceph monitoring, logging, and tracing, which addresses the technical problem of improving the overall observability, alarm capabilities, and distributed system fault diagnosis capabilities of Ceph clusters.

[0042] Example 1:

[0043] This invention provides an integrated method for Ceph monitoring, logging, and tracing, applied to a Ceph cluster. Each node in the Ceph cluster deploys OpenTelemetry Collector as the data acquisition layer, OpenTelemetry Protocol as the data transmission layer, OpenObserve as the data processing and display layer, and OpenObserve Tracing UI as the tracing visualization layer. The method includes four steps: data acquisition, data transmission, data processing and display, and tracing visualization.

[0044] Step S100 Data Acquisition: Collect monitoring metrics, log data, and link tracing data uniformly through OpenTelemetry Collector.

[0045] In this embodiment, the OpenTelemetry Collector obtains cluster status data in real time from Ceph Manager and collects log data in real time from Ceph OSD, Ceph MDS, and Ceph MON processes via file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it traces the complete execution path of client requests within the Ceph cluster, obtaining link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD, and MDS components. The OpenTelemetry Collector sends data in batch processing, compression, and retry mode, and supports local caching and automatic replay in the event of network interruption.

[0046] Step S200 Data transmission: Based on the OpenTelemetry protocol, monitoring metrics, log data, and link tracing data are uniformly sent to OpenObserver via the OpenTelemetry Protocol.

[0047] Step S300 Data Processing Demonstration: Within OpenObserve, monitoring metrics, log data, and trace data are written to the same database and a cascading index of trace_id and span_id is established. Through OpenObserve's built-in alarm engine, multi-level alarms are generated in real time based on log patterns, metric thresholds, and trace error rates, and a consumption interface is provided to ceph-mgr via a REST API.

[0048] In this embodiment, OpenObserve adopts a hybrid architecture of columnar storage and inverted index.

[0049] Step S400: Visualization of Link Tracing: The OpenObserve Tracing UI visualizes the complete call chain of client requests in CephMON, OSD, and MDS components, and correlates it with the monitoring metrics, log data, and link tracing data corresponding to the same trace_id.

[0050] The method in this embodiment has the following characteristics:

[0051] (1) Unified data collection architecture: Using OpenTelemetry Collector as a unified data collection platform, it simultaneously collects Ceph's monitoring metrics, log data, and trace data, eliminating the complexity of maintaining multiple systems;

[0052] (2) Real-time push mechanism: Configure OpenTelemetry Collector to obtain cluster status data from Ceph Manager (ceph-mgr) in real time, and collect logs from Ceph OSD, Ceph MDS and Ceph MON processes in real time through file monitoring;

[0053] (3) Distributed Link Tracing Integration: Utilizing OpenTelemetry's distributed tracing capabilities, the complete execution path of client requests within the Ceph cluster is traced, including the call chain from the client to components such as MON, OSD, and MDS;

[0054] (4) Integrated visualization solution: The collected metrics, logs and tracking data are sent to OpenObserve in a unified manner through the OpenTelemetry protocol, and its built-in visualization, analysis and link tracing functions are utilized without relying on third-party tools such as Grafana and Jaeger.

[0055] (5) Event-driven monitoring mode: Through real-time file monitoring and push mechanism, ensure that cluster status changes can be captured and processed in a timely manner, and avoid data omissions caused by scanning cycle;

[0056] (6) Correlation analysis capability: Through trace ID and span ID, the correlation query of metrics, logs and link data can be realized, providing end-to-end fault diagnosis capability.

[0057] This embodiment adopts a four-layer architecture design: data acquisition layer (OpenTelemetry Collector), data transmission layer (OpenTelemetry Protocol), data processing and display layer (OpenObserve), and link tracing visualization layer (OpenObserve Tracing UI).

[0058] Based on the method disclosed in this embodiment, the following operations are performed during the environment preparation stage:

[0059] (1) Deploy OpenTelemetry Collector on Ceph cluster nodes and ensure that it has permissions to access Ceph log files and management interfaces;

[0060] (2) Deploy an OpenObserver instance and configure the corresponding data receiving endpoint and storage backend.

[0061] Data acquisition configuration includes log acquisition configuration, data processing configuration, and data export configuration.

[0062] The relevant code for log collection configuration is as follows:

[0063]

[0064]

[0065]

[0066] The data processing configuration code is as follows:

[0067] The data export configuration code is as follows:

[0068]

[0069] The data pipeline configuration code is as follows:

[0070]

[0071] The high availability configuration code is as follows:

[0072] (1) Configure multiple OpenTelemetry Collector instances to achieve load balancing and failover;

[0073] (2) Set up data caching and retry mechanisms to ensure data integrity in the event of network anomalies.

[0074] The monitoring and alarm configurations are as follows:

[0075] (1) Configure real-time alarm rules in OpenObserver to trigger alarms based on log patterns and metric thresholds;

[0076] (2) Set up multi-level alarm policies, including three levels: warning, error and fatal.

[0077] The performance optimization configuration is as follows:

[0078] (1) Adjust the batch_size and timeout parameters according to the size of the Ceph cluster.

[0079] (2) Configure resource limits to prevent the Collector from consuming too many system resources.

[0080] The system verification and testing configuration is as follows:

[0081] Functional verification: Verify whether log and metric data are correctly collected and transmitted to OpenObserver, test the triggering and notification mechanism of alarm rules, and verify the data display effect of the visualization dashboard.

[0082] Performance testing: Test the data acquisition and transmission performance under high load conditions, and verify the system's stability and resource consumption.

[0083] Fault recovery test: Simulate abnormal situations such as network interruption and component failure to verify the system's self-recovery capability and data integrity protection mechanism.

[0084] The method in this embodiment realizes unified collection, real-time processing and intelligent analysis of Ceph monitoring and logs, which significantly improves the overall observability, alarm capability and operation and maintenance efficiency of Ceph clusters, and provides an innovative technical solution for the monitoring and management of distributed storage systems.

[0085] Example 2:

[0086] This invention provides an integrated system for Ceph monitoring, logging, and tracing, applicable to Ceph clusters. It includes OpenTelemetry Collector as the data acquisition layer, OpenTelemetry Protocol as the data transmission layer, OpenObserve as the data processing and display layer, and OpenObserve TracingUI as the tracing visualization layer.

[0087] OpenTelemetry Collector is used to collect monitoring metrics, log data, and link tracing data in a unified manner.

[0088] In this embodiment, the OpenTelemetry Collector obtains cluster status data in real time from Ceph Manager and collects log data in real time from Ceph OSD, Ceph MDS, and Ceph MON processes via file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it traces the complete execution path of client requests within the Ceph cluster, obtaining link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD, and MDS components. The OpenTelemetry Collector sends data in batch processing, compression, and retry mode, and supports local caching and automatic replay in the event of network interruption.

[0089] The OpenTelemetry Protocol is used to send monitoring metrics, log data, and tracing data to OpenObserver in a unified manner based on the OpenTelemetry protocol.

[0090] OpenObserve is used to write monitoring metrics, log data, and trace data to the same database and create cascading indexes of trace_id and span_id. It is used to generate multi-level alarms in real time based on log patterns, metric thresholds, and trace error rates through the built-in alarm engine, and provides a consumption interface to ceph-mgr through a REST API.

[0091] In this embodiment, OpenObserve adopts a hybrid architecture of columnar storage and inverted index.

[0092] The OpenObserve Tracing UI is used to visualize the complete call chain of client requests in Ceph MON, OSD, and MDS components, and to associate it with the monitoring metrics, log data, and tracing data corresponding to the same trace_id.

[0093] The system in this embodiment can execute the method disclosed in Embodiment 1 to achieve integrated monitoring, logging, and link tracing.

[0094] The above provides a detailed description of the integrated method and system for Ceph monitoring, logging, and link tracing provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An integrated method for Ceph monitoring, logging, and tracing, characterized in that, Applied to a Ceph cluster, each node of the Ceph cluster deploys OpenTelemetry Collector as the data acquisition layer, OpenTelemetry Protocol as the data transmission layer, OpenObserve as the data processing and display layer, and OpenObserve Tracing UI as the link tracing visualization layer. The process includes the following steps: Data Acquisition: Monitoring metrics, log data, and link tracing data are collected uniformly through the OpenTelemetry Collector; Data transmission: Based on the OpenTelemetry protocol, monitoring metrics, log data, and tracing data are uniformly sent to OpenObserver via the OpenTelemetry Protocol; Data processing and display: Within OpenObserve, monitoring metrics, log data, and trace data are written to the same database and a cascading index of trace_id and span_id is created. Through OpenObserve's built-in alarm engine, multi-level alarms are generated in real time based on log patterns, metric thresholds, and trace error rates, and a consumption interface is provided to ceph-mgr via a REST API. Link tracing visualization: The OpenObserve Tracing UI visualizes the complete call chain of client requests in Ceph MON, OSD, and MDS components, and correlates it with the monitoring metrics, log data, and link tracing data corresponding to the same trace_id.

2. The integrated method for Ceph monitoring, logging, and link tracing according to claim 1, characterized in that, The OpenTelemetry Collector obtains cluster status data from Ceph Manager in real time and collects log data from Ceph OSD, Ceph MDS, and Ceph MON processes in real time through file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it traces the complete execution path of client requests within the Ceph cluster to obtain link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD, and MDS components.

3. The integrated method for Ceph monitoring, logging, and link tracing according to claim 1, characterized in that, OpenTelemetry Collector sends data in a batch processing, compression, and retry manner, and supports local caching and automatic replay when the network is interrupted.

4. The integrated method for Ceph monitoring, logging, and link tracing according to claim 1, characterized in that, OpenObserve uses a hybrid architecture of columnar storage and inverted index.

5. An integrated system for Ceph monitoring, logging, and tracing, characterized in that, Applied to Ceph clusters, it includes OpenTelemetry Collector as the data acquisition layer, OpenTelemetryProtocol as the data transmission layer, OpenObserve as the data processing and display layer, and OpenObserveTracing UI as the link tracing visualization layer. OpenTelemetry Collector is used to uniformly collect monitoring metrics, log data, and link tracing data; The OpenTelemetry Protocol is used to uniformly send monitoring metrics, log data, and link tracing data to OpenObserver based on the OpenTelemetry protocol; OpenObserve is used to write monitoring metrics, log data, and trace data to the same database and create cascading indexes of trace_id and span_id. It is used to generate multi-level alarms in real time based on log patterns, metric thresholds, and trace error rates through the built-in alarm engine, and provides a consumption interface to ceph-mgr through a REST API. The OpenObserve Tracing UI is used to visualize the complete call chain of client requests in Ceph MON, OSD, and MDS components, and to associate it with the monitoring metrics, log data, and tracing data corresponding to the same trace_id.

6. The Ceph monitoring, logging, and link tracing integrated system according to claim 5, characterized in that, The OpenTelemetry Collector is used to obtain cluster status data from Ceph Manager in real time and collect log data from Ceph OSD, Ceph MDS, and Ceph MON processes in real time through file monitoring. Based on OpenTelemetry's distributed tracing capabilities, it traces the complete execution path of client requests within the Ceph cluster to obtain link tracing data. The complete execution path of client requests within the Ceph cluster includes the call chain from the client to the MON, OSD, and MDS components.

7. The Ceph monitoring, logging, and link tracing integrated system according to claim 5, characterized in that, OpenTelemetry Collector is used to send data in a batch processing + compression + retry manner, and supports local caching and automatic replay when the network is interrupted.

8. The Ceph monitoring, logging, and link tracing integrated system according to claim 5, characterized in that, OpenObserve uses a hybrid architecture of columnar storage and inverted index.

Images