A monitoring operation and maintenance system and implementation method for global acquisition integration

Abstract

The application discloses a kind of global acquisition integration's monitoring operation and maintenance system and implementation method, it is related to cloud computing technology field, the monitoring operation and maintenance system includes acquisition layer, transmission processing layer, storage layer and analysis layer, this kind of global acquisition integration's monitoring operation and maintenance system and implementation method, through the system, realize K8s-node, VM application, host, middleware and the full-dimensional acquisition of multiple types IT components, cover the complete monitoring chain from infrastructure to application layer, based on OTEL agreement realizes the full-process standardization of data acquisition, transmission, processing, eliminates the data format barrier between heterogeneous components, lays foundation for cross-component correlation analysis, through the hierarchical processing architecture of otelcol, in combination with near-end aggregation, data cleaning and other mechanisms, greatly improve data processing efficiency, reduce the stress of central node, classification storage strategy, select Prometheus and the adaptability storage scheme of Elasticsearch, give consideration to storage cost and query performance.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, specifically to a comprehensive monitoring and maintenance system and its implementation method that integrates global data acquisition. Background Technology

[0002] With the development of cloud computing and containerization technologies, IT system architectures are characterized by heterogeneous multi-component architectures and diverse deployment forms. Typical scenarios include hybrid deployments of Kubernetes container clusters, virtual machine (VM) application clusters, physical host clusters, and various middleware (such as message queues, databases, and caches).

[0003] Traditional monitoring solutions suffer from the following drawbacks: Limited data collection dimensions: Most solutions only target a single type of component (such as monitoring only Kubernetes or only hosts), failing to cover all IT resources; Fragmented data protocols: Different components use custom collection protocols (such as ZabbixAgent or custom APIs), resulting in inconsistent data formats and making cross-component correlation analysis difficult; Disjointed processing and storage: After data collection, there is a lack of unified processing logic, and storage media selection is chaotic (such as mixed storage of metrics and logs), increasing the complexity of data management and analysis; Isolated analysis perspective: Operations personnel need to switch between multiple tools to view metrics, links, and logs, making it impossible to form a comprehensive correlation analysis of "metrics-links-logs," resulting in low efficiency in troubleshooting. Summary of the Invention

[0004] The purpose of this invention is to provide a monitoring and maintenance system and implementation method that integrates global data acquisition, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a monitoring and maintenance system integrating full-domain data acquisition, wherein the monitoring and maintenance system includes an acquisition layer, a transmission and processing layer, a storage layer and an analysis layer; The acquisition layer includes a multi-source heterogeneous acquisition module, which deploys dedicated OTELCollector acquisition components for different types of monitoring targets to achieve full-dimensional data acquisition. The transmission processing layer includes a gateway and an OTEL processing module. The gateway and OTEL processing module are responsible for ensuring the reliable transmission of data from the acquisition layer and performing preliminary processing on the data to provide high-quality and standardized data for subsequent storage and analysis. The storage layer includes a categorized storage module, which adopts a "metric-link-log" categorized storage strategy and selects an adaptive storage medium. The analysis layer includes a panoramic analysis module, which, based on the data provided by the storage layer, provides operation and maintenance personnel with a comprehensive, in-depth, and intuitive insight into the system's operational status, assisting them in decision-making and problem-solving.

[0006] Preferably, the multi-source heterogeneous acquisition module includes: K8s-Node Collection Unit: Through the K8s API and container runtime interface, it collects node resource utilization, Pod lifecycle and container performance metrics, and encapsulates the data into OTEL protocol output by the built-in OTELCollector; VM Application Collection Unit: Deploys OTELCollector and otelagent within the virtual machine to collect JVM / process performance, business metrics, and call chain data of the application; Host Metrics Collection Unit: Deploy a lightweight OTELCollector on a physical / virtual host to collect the host's hardware metrics and operating system status; Middleware collection unit: For database, message queue and cache middleware, it collects operation metrics through their dedicated APIs or client tools, and outputs them after conversion by OTELCollector unified protocol.

[0007] Preferably, the gateway and OTEL processing module includes a gateway submodule and an OTEL processing submodule: Gateway submodule: As the communication hub between the acquisition layer and the transmission processing layer, it supports multi-protocol access, performs preliminary data verification, flow control and routing, and ensures that the acquired data is reliably transmitted to the OTEL processing module; The OTEL processing submodule includes: Receiving Unit: as the central Collector, otelcol is responsible for receiving various OTEL data forwarded by the gateway. As an edge Collector, otelcol can be deployed at the network edge node to preprocess near-end data before transmitting it to otelcol, reducing the pressure on the central node. Link and Log Generation Unit: Generates a visual link topology based on received Trace data, generates structured log entries based on log event data, and supports the association tagging of links and logs.

[0008] Preferably, the classification storage module includes: Application metrics library: It uses the Prometheus database to store time-series metrics data for K8s nodes, VM applications and hosts, and supports high-concurrency writing and multi-dimensional aggregation queries; Middleware metrics library: It uses the Prometheus database to store the performance and health metrics of middleware, and is logically linked to the application metrics library through tags; Link Library: Uses Elasticsearch database to store Span data of call links, and supports complex retrieval and link topology analysis based on service, method, and latency; Log library: Uses Elasticsearch database to store structured log data, supports fast retrieval by time, level, service and keywords, and associates it with link data through TraceID.

[0009] Preferably, the panoramic analysis module includes: Visual dashboards: Based on the Grafana tool, create multi-dimensional dashboards that support trend analysis and comparative analysis of indicators; Link tracing analysis: Display service call relationships through link topology diagrams, locate delayed nodes and faulty links, and achieve "link-log" linkage troubleshooting by combining log data; Intelligent Alarms: Based on PrometheusAlertmanager or a self-developed alarm engine, it allows setting up multi-level alarm rules and supports alarm notifications via email, SMS, and instant messaging tools; Fault Root Cause Analysis: By correlating abnormal indicators, link delays, and log error source data, the system automatically generates a fault root cause analysis report to help operations and maintenance personnel quickly locate and resolve problems.

[0010] A method for implementing a comprehensive monitoring and maintenance system that integrates data acquisition across the entire domain, the method comprising the following steps: S1: Multi-source data acquisition steps: S1.1: For K8s nodes, deploy OTELCollector in the form of DaemonSet within the cluster. Collect node and container metrics through kubeletAPI and cAdvisor. Collect resource utilization every 30 seconds, collect Pod events in real time, and output the data in OTELMetrics and Traces formats. S1.2: For VM applications, install the OTELCollector binary package inside the virtual machine and configure the OTELCollector probe to collect data when the application starts; S1.3: For host metrics, install the lightweight OTELCollector on the host to collect basic metrics of CPU, memory and disk, with a collection interval of 15 seconds. S1.4: For middleware, taking the database MySQL as an example, deploy the MySQLexporter plugin of OTELCollector to collect indicators such as connection number, query latency and transaction success rate, with a collection interval of 1 minute. The message queue Kafka collects indicators such as partition number and production / consumption rate through its JMX interface. S2: Standardized data transmission steps: OTEL data output from the acquisition layer is sent to the gateway module via the gRPC protocol. After the gateway performs format verification and flow control on the data, it forwards it to the otelcol or otelcol component of the OTEL processing module. For edge nodes deployed across regions, otelcol can first perform near-end aggregation, and then transmit the aggregated data to the central otelcol to reduce network transmission overhead. S3: Data Processing and Generation Steps: S3.1: After receiving data, telcol performs data cleaning, indicator aggregation, and link association through its built-in processor; S3.2: The link and log generation unit generates a visualized link topology based on the processed Trace data. At the same time, it converts unstructured logs into structured data through a log processor and injects TraceID to realize the association between links and logs. S4: Categorized storage steps: S4.1: Application metrics and middleware metrics data: written to the corresponding Prometheus database through the Prometheus RemoteWrite interface, and sharded according to "service name-cluster name-metric type" during storage to ensure query performance; S4.2: Linked Data: Write Span data to Elasticsearch in JSON format, divide the index by day, and set reasonable shards and replicas to ensure data reliability and retrieval efficiency; S4.3: Log data: Structured logs are written to Elasticsearch in JSON format. The index is also divided by day, and index templates are set based on log level and service name to achieve fine-grained storage management. S5: Panoramic Analysis and Operation / Maintenance Steps: S5.1: Visual Analysis: Configure data sources in Grafana to create multi-dimensional dashboards that display node CPU / memory usage and Pod running status; the "Application Performance Dashboard" displays trends in interface response time, throughput, and error rate. S5.2: Link Tracing: Retrieve the complete call chain in the panoramic observation platform by link ID, analyze the latency ratio of each service node, and locate performance bottlenecks. S5.3: Set alarm rules, and push alarm details via WeChat Work after an alarm is triggered, including indicator trend charts and related log snippets; S5.4: When an alarm is received, the platform automatically associates the call chain of the interface, the resource indicators of the node, the system logs and application logs of the node, and generates a root cause analysis report.

[0011] Preferably, in step S1.2, the data acquisition metric for the otelagent probe is configured to be once per minute, with real-time data acquisition across the link.

[0012] Preferably, the format verification in step S2 is to check the integrity of the OTEL protocol fields, and the flow control is based on the token bucket algorithm to limit the number of concurrent requests.

[0013] Preferably, step S3.2: data cleaning includes filtering invalid tags and completing missing fields.

[0014] Preferably, the warning rules in step S5.3 are host CPU utilization > 80% for 5 consecutive minutes and application interface error rate > 5%.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention enables comprehensive data collection from various IT components, including Kubernetes nodes, VM applications, hosts, and middleware, covering the entire monitoring chain from infrastructure to application layer. Based on the OTEL protocol, it standardizes the entire process of data collection, transmission, and processing, eliminating data format barriers between heterogeneous components and laying the foundation for cross-component correlation analysis. Through OTEL's layered processing architecture, combined with near-end aggregation and data cleaning mechanisms, it significantly improves data processing efficiency, reduces the pressure on central nodes, and employs categorized storage strategies, selecting compatible storage solutions for Prometheus and Elasticsearch to balance storage costs and query performance. Furthermore, through visualization, alerting, and fault tracing capabilities, it significantly shortens troubleshooting time. Attached Figure Description

[0016] Figure 1 This is a diagram of the overall monitoring architecture for integrated full-domain data acquisition according to the present invention; In the diagram: 1. Kubernetes node; 2. VM application; 3. Host; 4. Middleware; 5. Metrics collector; 6. Link & log collector; 7. Database; 8. Analysis platform. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example: Please refer to Figure 1 The present invention provides a technical solution: a monitoring and maintenance system integrating full-domain data acquisition, the monitoring and maintenance system including a data acquisition layer, a transmission and processing layer, a storage layer and an analysis layer; The acquisition layer includes a multi-source heterogeneous acquisition module. The multi-source heterogeneous acquisition module deploys dedicated OTELCollector acquisition components for different types of monitoring targets to achieve full-dimensional data acquisition. The transmission processing layer includes a gateway and an OTEL processing module. The gateway and OTEL processing module are responsible for ensuring the reliable transmission of data from the acquisition layer and performing preliminary processing on the data to provide high-quality, standardized data for subsequent storage and analysis. The storage layer includes a categorized storage module, which adopts a "metric-link-log" categorized storage strategy and selects an adaptive storage medium. The analysis layer includes a panoramic analysis module, which, based on the data provided by the storage layer, provides operations and maintenance personnel with a comprehensive, in-depth, and intuitive understanding of the system's operational status, assisting them in decision-making and problem-solving.

[0019] The multi-source heterogeneous acquisition module includes: K8s-Node Collection Unit: Through the K8s API and container runtime interface, it collects node resource utilization, Pod lifecycle and container performance metrics, and encapsulates the data into OTEL protocol output by the built-in OTELCollector; VM Application Collection Unit: Deploys OTELCollector and otelagent within the virtual machine to collect JVM / process performance, business metrics, and call chain data of the application; Host Metrics Collection Unit: Deploy a lightweight OTELCollector on a physical / virtual host to collect the host's hardware metrics and operating system status; Middleware collection unit: For database, message queue and cache middleware, it collects operation metrics through their dedicated APIs or client tools, and outputs them after conversion by OTELCollector unified protocol.

[0020] The gateway and OTEL processing module includes a gateway submodule and an OTEL processing submodule: Gateway submodule: As the communication hub between the acquisition layer and the transmission processing layer, it supports multi-protocol access, performs preliminary data verification, flow control and routing, and ensures that the acquired data is reliably transmitted to the OTEL processing module; The OTEL processing submodule includes: Receiving Unit: as the central Collector, otelcol is responsible for receiving various OTEL data forwarded by the gateway. As an edge Collector, otelcol can be deployed at the network edge node to preprocess near-end data before transmitting it to otelcol, reducing the pressure on the central node. Link and Log Generation Unit: Generates a visual link topology based on received Trace data, generates structured log entries based on log event data, and supports the association tagging of links and logs.

[0021] The categorized storage module includes: Application metrics library: It uses the Prometheus database to store time-series metrics data for K8s nodes, VM applications and hosts, and supports high-concurrency writing and multi-dimensional aggregation queries; Middleware metrics library: It uses the Prometheus database to store the performance and health metrics of middleware, and is logically linked to the application metrics library through tags; Link Library: Uses Elasticsearch database to store Span data of call links, and supports complex retrieval and link topology analysis based on service, method, and latency; Log library: Uses Elasticsearch database to store structured log data, supports fast retrieval by time, level, service and keywords, and associates it with link data through TraceID.

[0022] The panoramic analysis module includes: Visual dashboards: Based on the Grafana tool, create multi-dimensional dashboards that support trend analysis and comparative analysis of indicators; Link tracing analysis: Display service call relationships through link topology diagrams, locate delayed nodes and faulty links, and achieve "link-log" linkage troubleshooting by combining log data; Intelligent Alarms: Based on PrometheusAlertmanager or a self-developed alarm engine, it allows setting up multi-level alarm rules and supports alarm notifications via email, SMS, and instant messaging tools; Fault Root Cause Analysis: By correlating abnormal indicators, link delays, and log error source data, the system automatically generates a fault root cause analysis report to help operations and maintenance personnel quickly locate and resolve problems.

[0023] The specific implementation method of this embodiment is as follows: Acquisition layer: K8s Node Acquisition Unit: Interacts with the K8s cluster using the K8s API and container runtime interface. Through these interfaces, it can obtain node-level resource utilization information, such as CPU utilization and memory usage. It can also monitor the Pod lifecycle, including Pod creation, startup, and shutdown status changes. Furthermore, it can collect container performance metrics, such as container CPU consumption and memory usage. The built-in OTELCollector encapsulates the collected data according to the OTEL protocol to ensure data format uniformity and standardization before outputting it to the subsequent transmission and processing layer. VM Application Data Acquisition Unit: OTELCollector and otelagent are deployed within the virtual machine. otelagent, acting as an application probe, is embedded in the application process and can collect JVM performance metrics such as JVM memory usage and garbage collection status; it can also collect process performance metrics such as CPU utilization and memory usage, as well as business metrics such as application interface response time and processing volume; and call chain data, recording the call relationships and time consumption between various methods within the application. The collected data is processed uniformly by OTELCollector and output in OTEL protocol format. Host metrics collection unit: Deploy a lightweight OTELCollector on a physical host or virtual host. It mainly collects the host's hardware metrics, such as CPU model, number of cores, frequency, memory size and type, disk capacity and usage, etc. At the same time, it obtains the operating system's status information, including the operating system version, runtime, system load, etc. The collected data is encapsulated into the OTEL protocol format and sent to the transmission processing layer. Middleware Data Collection Unit: For different types of middleware, such as database, message queue, and caching middleware, use their dedicated APIs or client tools to collect operational metrics. For example, for databases, metrics such as connection count, query latency, and transaction success rate can be collected; for message queues, metrics such as partition count and production / consumption rate can be collected.

[0024] Transport processing layer: The gateway submodule serves as the communication hub between the acquisition layer and the processing layer, supporting multi-protocol access (OTELgRPC, HTTP), and performing preliminary data verification, flow control, and routing to ensure reliable transmission of acquired data to the OTEL processing module. The receiving unit of the OTEL processing submodule is the otelcol component. As the central collector, otelcol is responsible for receiving various types of OTEL data forwarded by the gateway. As an edge collector, otelcol can be deployed at network edge nodes (such as regional nodes in a multi-cloud environment) to perform near-end data preprocessing (such as aggregation and filtering) before transmitting it to otelcol, thereby reducing the pressure on the central node. Link and Log Generation Unit: Generates a visual link topology based on the received Trace data (call link), generates structured log entries based on log event data (such as application logs and system logs), and supports the association tagging of links and logs (such as association via TraceID).

[0025] Storage layer: Application metrics library: It uses the Prometheus database to store time-series metrics data for K8s nodes, VM applications, and hosts, and supports high-concurrency writing and multi-dimensional aggregation queries; Middleware metrics library: It uses the Prometheus database to store the performance and health metrics of middleware, and is logically associated with the application metrics library through tags (such as service name and cluster name); Link Library: Uses Elasticsearch database to store Span data of call links, and supports complex retrieval and link topology analysis based on service, method, and latency; Log library: Uses Elasticsearch database to store structured log data, supports fast retrieval by time, level, service, and keywords, and associates it with link data through TraceID.

[0026] Analysis layer: Visual dashboards: Based on tools such as Grafana, create multi-dimensional dashboards (such as K8s resource overview, application performance dashboard, middleware health dashboard), supporting trend analysis and comparative analysis of indicators; Link tracing analysis: Display service call relationships through link topology diagrams, locate delayed nodes and faulty links, and achieve "link-log" linkage troubleshooting by combining log data; Intelligent Alarms: Based on PrometheusAlertmanager or a self-developed alarm engine, it allows setting multi-level alarm rules (such as resource utilization thresholds and service availability thresholds) and supports alarm notifications via email, SMS, and instant messaging tools (such as WeChat Work and Slack). Fault tracing and analysis: By correlating abnormal indicators, link delays, log errors and other multi-source data, the system automatically generates a fault root cause analysis report to help maintenance personnel quickly locate and resolve problems.

[0027] A method for implementing a comprehensive monitoring and maintenance system that integrates data acquisition across the entire domain includes the following steps: S1: Multi-source data acquisition steps: S1.1: For K8s nodes, deploy OTELCollector in the form of DaemonSet within the cluster. Collect node and container metrics through kubeletAPI and cAdvisor. Collect resource utilization every 30 seconds, collect Pod events in real time, and output the data in OTELMetrics and Traces formats. S1.2: For VM applications, install the OTELCollector binary package inside the virtual machine and configure the OTELCollector probe to collect data when the application starts; S1.3: For host metrics, install the lightweight OTELCollector on the host to collect basic metrics of CPU, memory and disk, with a collection interval of 15 seconds. S1.4: For middleware, taking the database MySQL as an example, deploy the MySQLexporter plugin of OTELCollector to collect indicators such as connection number, query latency and transaction success rate, with a collection interval of 1 minute. The message queue Kafka collects indicators such as partition number and production / consumption rate through its JMX interface. S2: Standardized data transmission steps: OTEL data output from the acquisition layer is sent to the gateway module via the gRPC protocol. After the gateway performs format verification and flow control on the data, it forwards it to the otelcol or otelcol component of the OTEL processing module. For edge nodes deployed across regions, otelcol can first perform near-end aggregation, and then transmit the aggregated data to the central otelcol to reduce network transmission overhead. S3: Data Processing and Generation Steps: S3.1: After receiving data, telcol performs data cleaning, indicator aggregation, and link association through its built-in processor; S3.2: The link and log generation unit generates a visualized link topology based on the processed Trace data. At the same time, it converts unstructured logs into structured data through a log processor and injects TraceID to realize the association between links and logs. S4: Categorized storage steps: S4.1: Application metrics and middleware metrics data: written to the corresponding Prometheus database through the Prometheus RemoteWrite interface, and sharded according to "service name-cluster name-metric type" during storage to ensure query performance; S4.2: Linked Data: Write Span data to Elasticsearch in JSON format, divide the index by day, and set reasonable shards and replicas to ensure data reliability and retrieval efficiency; S4.3: Log data: Structured logs are written to Elasticsearch in JSON format. The index is also divided by day, and index templates are set based on log level and service name to achieve fine-grained storage management. S5: Panoramic Analysis and Operation / Maintenance Steps: S5.1: Visual Analysis: Configure data sources in Grafana to create multi-dimensional dashboards that display node CPU / memory usage and Pod running status; the "Application Performance Dashboard" displays trends in interface response time, throughput, and error rate. S5.2: Link Tracing: Retrieve the complete call chain in the panoramic observation platform by link ID, analyze the latency ratio of each service node, and locate performance bottlenecks. S5.3: Set alarm rules, and push alarm details via WeChat Work after an alarm is triggered, including indicator trend charts and related log snippets; S5.4: When an alarm is received, the platform automatically associates the call chain of the interface, the resource indicators of the node, the system logs and application logs of the node, and generates a root cause analysis report.

[0028] In step S1.2, the otelagent probe is configured to collect data once per minute, with real-time data collection across the link.

[0029] The format validation in step S2 checks the integrity of the OTEL protocol fields, and the flow control limits the number of concurrent requests based on the token bucket algorithm.

[0030] Step S3.2: Data cleaning includes filtering invalid labels and completing missing fields.

[0031] The warning rules in step S5.3 are: host CPU utilization > 80% for 5 consecutive minutes and application interface error rate > 5%.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0033] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A comprehensive monitoring and maintenance system integrating data acquisition, characterized in that: The monitoring and maintenance system includes a data acquisition layer, a transmission and processing layer, a storage layer, and an analysis layer. The acquisition layer includes a multi-source heterogeneous acquisition module, which deploys dedicated OTELCollector acquisition components for different types of monitoring targets to achieve full-dimensional data acquisition. The transmission processing layer includes a gateway and an OTEL processing module. The gateway and OTEL processing module are responsible for ensuring the reliable transmission of data from the acquisition layer and performing preliminary processing on the data to provide high-quality and standardized data for subsequent storage and analysis. The storage layer includes a categorized storage module, which adopts an indicator-link-log categorized storage strategy and selects an adaptive storage medium. The analysis layer includes a panoramic analysis module, which provides system operation and maintenance personnel with insights into the system's operational status based on the data provided by the storage layer, assisting them in decision-making and problem handling.

2. The integrated monitoring and maintenance system for full-domain data acquisition according to claim 1, characterized in that: The multi-source heterogeneous acquisition module includes: K8s-Node Collection Unit: Through the K8s API and container runtime interface, it collects node resource utilization, Pod lifecycle and container performance metrics, and encapsulates the data into OTEL protocol output by the built-in OTELCollector; VM Application Collection Unit: Deploys OTELCollector and otelagent within the virtual machine to collect JVM / process performance, business metrics, and call chain data of the application; Host Metrics Collection Unit: Deploy a lightweight OTELCollector on a physical / virtual host to collect the host's hardware metrics and operating system status; Middleware collection unit: For database, message queue and cache middleware, it collects operation metrics through their dedicated APIs or client tools, and outputs them after conversion by OTELCollector unified protocol.

3. The integrated monitoring and maintenance system for full-domain data acquisition according to claim 1, characterized in that: The gateway and OTEL processing module includes a gateway submodule and an OTEL processing submodule: Gateway submodule: As the communication hub between the acquisition layer and the transmission processing layer, it supports multi-protocol access, performs preliminary data verification, flow control and routing, and ensures that the acquired data is reliably transmitted to the OTEL processing module; The OTEL processing submodule includes: Receiving Unit: as the central Collector, otelcol is responsible for receiving various OTEL data forwarded by the gateway. As an edge Collector, otelcol can be deployed at the network edge node to preprocess near-end data before transmitting it to otelcol, reducing the pressure on the central node. Link and Log Generation Unit: Generates a visual link topology based on received Trace data, generates structured log entries based on log event data, and supports the association tagging of links and logs.

4. The integrated monitoring and maintenance system for full-domain data acquisition according to claim 1, characterized in that: The classification storage module includes: Application metrics library: It uses the Prometheus database to store time-series metrics data for K8s nodes, VM applications and hosts, and supports high-concurrency writing and multi-dimensional aggregation queries; Middleware metrics library: It uses the Prometheus database to store the performance and health metrics of middleware, and is logically linked to the application metrics library through tags; Link Library: Uses Elasticsearch database to store Span data of call links, and supports complex retrieval and link topology analysis based on service, method, and latency; Log library: Uses Elasticsearch database to store structured log data, supports fast retrieval by time, level, service and keywords, and associates it with link data through TraceID.

5. The integrated monitoring and maintenance system for full-domain data acquisition according to claim 1, characterized in that: The panoramic analysis module includes: Visual dashboards: Based on the Grafana tool, create multi-dimensional dashboards that support trend analysis and comparative analysis of indicators; Link tracing and analysis module: Displays service call relationships through link topology diagrams, locates delayed nodes and faulty links, and combines log data to achieve link-log linkage troubleshooting; Intelligent alarm module: Based on PrometheusAlertmanager or a self-developed alarm engine, it allows setting up multi-level alarm rules and supports alarm notifications via email, SMS, and instant messaging tools; Fault tracing and analysis module: By associating abnormal indicators, link delays and log error source data, it automatically generates fault root cause analysis reports to help operation and maintenance personnel quickly locate and solve problems.

6. A method for implementing a comprehensive monitoring and maintenance system with integrated data acquisition, characterized in that: The implementation method includes the following steps: S1: Multi-source data acquisition steps: S1.1: For K8s nodes, deploy OTELCollector in the form of DaemonSet within the cluster. Collect node and container metrics through kubeletAPI and cAdvisor. Collect resource utilization every 30 seconds, collect Pod events in real time, and output the data in OTELMetrics and Traces formats. S1.2: For VM applications, install the OTELCollector binary package inside the virtual machine and configure the OTELCollector probe to collect data when the application starts; S1.3: For host metrics, install the lightweight OTELCollector on the host to collect basic metrics of CPU, memory and disk, with a collection interval of 15 seconds. S1.4: For middleware, taking the database MySQL as an example, deploy the MySQLexporter plugin of OTELCollector to collect indicators such as connection number, query latency and transaction success rate, with a collection interval of 1 minute. The message queue Kafka collects indicators such as partition number and production / consumption rate through its JMX interface. S2: Standardized data transmission steps: OTEL data output from the acquisition layer is sent to the gateway module via the gRPC protocol. After the gateway performs format verification and flow control on the data, it forwards it to the otelcol or otelcol component of the OTEL processing module. For edge nodes deployed across regions, otelcol can first perform near-end aggregation, and then transmit the aggregated data to the central otelcol to reduce network transmission overhead. S3: Data Processing and Generation Steps: S3.1: After receiving data, telcol performs data cleaning, indicator aggregation, and link association through its built-in processor; S3.2: The link and log generation unit generates a visualized link topology based on the processed Trace data. At the same time, it converts unstructured logs into structured data through a log processor and injects TraceID to realize the association between links and logs. S4: Categorized storage steps: S4.1: Application metrics and middleware metrics data: written to the corresponding Prometheus database through the Prometheus RemoteWrite interface, and sharded by service name-cluster name-metric type during storage to ensure query performance; S4.2: Linked Data: Write Span data to Elasticsearch in JSON format, divide the index by day, and set reasonable shards and replicas to ensure data reliability and retrieval efficiency; S4.3: Log data: Structured logs are written to Elasticsearch in JSON format. The index is also divided by day, and index templates are set based on log level and service name to achieve fine-grained storage management. S5: Panoramic Analysis and Operation / Maintenance Steps: S5.1: Visual Analysis: Configure data sources in Grafana to create multi-dimensional dashboards that display node CPU / memory usage and Pod running status; application performance dashboards display trends in interface response time, throughput, and error rate. S5.2: Link Tracing: Retrieve the complete call chain in the panoramic observation platform by link ID, analyze the latency ratio of each service node, and locate performance bottlenecks. S5.3: Set alarm rules, and push alarm details via WeChat Work after an alarm is triggered, including indicator trend charts and related log snippets; S5.4: When an alarm is received, the platform automatically associates the call chain of the interface, the resource indicators of the node, the system logs and application logs of the node, and generates a root cause analysis report.

7. The method for implementing a comprehensive monitoring and maintenance system based on claim 1, characterized in that: In step S1.2, the otelagent probe is configured to collect data once per minute, with real-time data collection across the link.

8. The method for implementing a comprehensive monitoring and maintenance system based on claim 1, characterized in that: The format verification in step S2 checks the integrity of the OTEL protocol fields, and the flow control limits the number of concurrent requests based on the token bucket algorithm.

9. The method for implementing a comprehensive monitoring and maintenance system based on claim 1, characterized in that: The data cleaning in step S3.2 includes filtering invalid labels and filling in missing fields.

10. The method for implementing a comprehensive monitoring and maintenance system based on claim 1, characterized in that: The warning rules in step S5.3 are: host CPU utilization > 80% for 5 consecutive minutes and application interface error rate > 5%.

Images