Distributed data state acquisition system

Abstract

The invention relates to the technical field of computer application, in particular to a distributed data state acquisition system, which comprises the following hierarchical modules: an edge layer, a transmission layer, a storage layer, a calculation layer and a control layer. A distributed data state acquisition system is characterized in that an eBPF program is directly mounted in a kernel space, events such as system calling, network flow and file operation are captured, the overhead of switching between a user mode and a kernel mode of a traditional Agent is bypassed, when the node load is low, an eBPF probe of a non-key index is automatically suspended, and only basic monitoring is reserved; the edge node caches the original data, only uploads an aggregation result after calculating a mean value and a quantile, and reduces 80% of transmission data volume; a filtering rule is loaded through a WASM module, invalid data processing is reduced by 70%, direct collection is carried out on a kernel layer based on an eBPF, and CPU occupation is smaller than 0.5%.

Description

Technical Field

[0001] The present invention relates to the technical field of computer applications, and in particular to a distributed data status acquisition system. Background Art

[0002] Distributed data processing is a method for processing large datasets by distributing data and computation across multiple nodes, achieving high performance and scalability. With the advent of the big data era, distributed data processing technology has become the mainstream method for processing large amounts of data.

[0003] Data status collection mainly collects the following information: node status: CPU, memory, disk, and network usage; service status: service availability, response time, and error rate (such as HTTP 500); data status: distributed database / storage replica consistency, shard load, and transaction latency; message queue: backlog and throughput (such as Kafka's Lag); and custom metrics: key business-related metrics. Existing distributed data collection systems (such as Prometheus and the ELK technology stack) have the following defects. For example, traditional agents (such as Node Exporter) need to initiate system calls for each metric, and frequent context switching causes the CPU to occupy more than 5%.

[0004] In order to solve the above problems, we make improvements and propose a distributed data status acquisition system. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: The present invention provides a distributed data status acquisition system, comprising the following hierarchical modules: Edge layer: Deployed on distributed nodes, it includes a lightweight collection agent based on eBPF technology, which is used to non-invasively collect system metrics and perform adaptive sampling and local data preprocessing. Transport layer: supports dual-mode transmission protocols, including real-time channels (gRPC + Protocol Buffers) and batch channels (Kafka + Avro), and hierarchical transmission based on data priority; Storage layer: uses a multi-level storage engine, including time series database (hot storage), column storage (warm storage), and object storage (cold storage), and dynamically shards based on business tags; Computing layer: Integrates streaming computing framework and batch processing engine to support real-time event processing and offline data analysis; Control layer: Implements federated coordination based on the Raft protocol, dynamically manages node registration, load balancing, and scaling.

[0006] As a preferred technical solution of the present invention, the edge layer further includes: an embedded rule engine (based on a WASM module) for performing pre-filtering before data is uploaded. The filtering rules include ignoring indicators with specified tags or removing duplicate logs; a local cache module for aggregating raw data using a sliding time window to generate mean, quantile or standard deviation statistics.

[0007] As a preferred technical solution of the present invention, the priority queue of the transport layer is implemented as follows: data is divided into three levels: key events, performance indicators and regular logs, among which key events (such as service downtime and error rate exceeding the threshold) give priority to occupying real-time channel bandwidth; and the weighted round-robin algorithm (WRR) is used to allocate transmission resources of batch channels.

[0008] As a preferred technical solution of the present invention, the dynamic sharding strategy of the storage layer includes: hashing shards based on business tags (region, service, env) to ensure that data of the same business dimension are physically stored adjacently; automatically triggering rebalancing based on shard load monitoring to migrate hot shards to low-load storage nodes.

[0009] As a preferred technical solution of the present invention, the system also includes a hybrid clock synchronization module, which deploys a hybrid logical clock (HLC) at the edge layer node, combines the physical clock and the logical counter to generate a global ordered timestamp; in the calibration phase, the reference time is aligned through the NTP protocol, with an error range of less than 1 millisecond.

[0010] As a preferred technical solution of the present invention, the system adopts an adaptive compression algorithm, including: using the Gorilla compression algorithm for time series data to delete redundant timestamp bytes of continuous data; using the Zstandard algorithm for log text data to dynamically adjust the compression level (level 1-22) and select the balance point between compression speed and compression rate according to CPU load.

[0011] As a preferred technical solution of the present invention, the federated coordinator of the control layer implements: electing a master coordinating node through the Raft protocol, responsible for the persistence and synchronization of cluster metadata (node ​​status, shard mapping table); when a node is overloaded, predicting the optimal scaling strategy based on the Q-learning algorithm, and automatically scheduling the start and stop of the edge layer collection agent.

[0012] The beneficial effects of the present invention are: a distributed data status acquisition system that directly mounts eBPF programs in kernel space, captures events such as system calls, network traffic, file operations, etc., bypasses the overhead of traditional agents switching between user mode and kernel mode, and automatically suspends eBPF probes for non-critical indicators when the node load is low, retaining only basic monitoring; edge nodes cache raw data, and only upload aggregated results after calculating the mean and quantile, reducing the amount of transmitted data by 80%; filtering rules are loaded through the WASM module, reducing invalid data processing by 70%, and based on direct collection at the kernel layer based on eBPF, the CPU occupancy is less than 0.5%. DETAILED DESCRIPTION

[0013] The preferred embodiments of the present invention are described below. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0014] Embodiment: A distributed data status acquisition system includes the following hierarchical modules: Edge layer: Deployed on distributed nodes, it includes a lightweight collection agent based on eBPF technology for non-intrusive collection of system metrics, as well as adaptive sampling and local data preprocessing. eBPF non-intrusive collection: eBPF programs are directly mounted in kernel space to capture events such as system calls, network traffic, and file operations, bypassing the overhead of traditional agents switching between user and kernel mode. For example, when collecting network metrics, eBPF filters packets in the kernel layer via XDP (eXpress Data Path), eliminating the need to copy them to user mode. A dynamic offloading mechanism: When node load is low, eBPF probes for non-critical metrics (such as disk IO) are automatically paused, retaining only basic monitoring. Sliding window aggregation: Edge nodes cache raw data (e.g., a 5-second window) and only upload aggregated results after calculating the mean and quantile, reducing data transmission by 80%. Rule engine filtering: Filtering rules are loaded through the WASM module (e.g., discarding redundant data with CPU utilization <30% and no error logs), reducing invalid data processing by 70%. Transport layer: Supports dual-mode transmission protocols, including real-time channels (gRPC + Protocol Buffers) and batch channels (Kafka + Avro), and hierarchical transmission based on data priority, transmission encryption, full-link TLS 1.3 encryption, forward secrecy (PFS) and AEAD encryption algorithms (such as AES-GCM); Storage layer: A multi-level storage engine is used, including a time series database (hot storage), columnar storage (warm storage), and object storage (cold storage). Dynamic sharding is performed based on business tags. Hot storage retains the last 7 days of raw data, warm storage retains 30 days of downsampled data, and cold storage stores archived historical data in Parquet format. Adaptive compression is used. For time series data, the Gorilla algorithm removes redundant timestamps (only stores differences) and floating-point numbers (XOR encoding). For log data, Zstandard dynamically selects the compression level (high compression ratio under low load and fast compression under high load). Computing layer: Integrates streaming computing framework and batch processing engine to support real-time event processing and offline data analysis; Control layer: Implements federated coordination based on the Raft protocol, dynamically manages node registration, load balancing, and scaling. The control layer elects the leader node, and persists metadata (shard mapping, node status) to the majority node. When the leader fails, the remaining nodes re-elect in less than 1 second to ensure metadata availability. Load trends are predicted based on the Q-learning algorithm. Historical load, time period, and business tags are input to output the optimal edge proxy start and stop policy.

[0015] Preferably, the edge layer further includes: an embedded rule engine (based on a WASM module) for performing pre-filtering before data is uploaded, wherein the filtering rules include ignoring indicators with specified tags or eliminating duplicate logs; a local cache module for aggregating raw data with a sliding time window (5 to 60 seconds), generating mean, quantile or standard deviation statistics, and dynamically adjusting the collection frequency. Basic indicators (CPU, memory) are collected every 10 seconds when the load is low, and full indicators are collected every second when the load is high, and real-time channel transmission is triggered; the real-time channel uses gRPC streaming transmission, combined with Protocol Buffers binary encoding, with an end-to-end delay of less than 50ms, Flink window calculation, real-time anomaly detection, edge nodes are grouped by geographic location, and each group independently summarizes data to the central layer to avoid global single-point bottlenecks.

[0016] Preferably, the priority queue implementation method of the transport layer is as follows: data is divided into three levels: key events, performance indicators and regular logs, among which key events (such as service downtime and error rate exceeding the threshold) give priority to occupying real-time channel bandwidth; and the weighted round-robin algorithm (WRR) is used to allocate transmission resources of batch channels.

[0017] The optimal dynamic sharding strategy for the storage layer includes: hashing shards based on business tags (region, service, env) to ensure that data in the same business dimension is physically stored adjacently; automatically triggering rebalancing based on shard load monitoring to migrate hot shards to low-load storage nodes, and adopting a federated sharding architecture: Horizontal expansion: Each shard independently processes data with a specific business tag (such as region=us-east), and the number of shards can be dynamically increased with the node scale. Dynamic rebalancing monitors shard load (such as QPS and storage capacity). When the load of a single shard exceeds the threshold, it is automatically split into two sub-shards and migrated to a lower-load node.

[0018] Furthermore, the system also includes a hybrid clock synchronization module, which deploys a hybrid logical clock (HLC) on edge nodes, combining physical clocks and logical counters to generate globally ordered timestamps. During the calibration phase, the NTP protocol is used to align the reference time with an error range of less than 1 millisecond. The timestamp of each event consists of two parts: Physical time: local clock synchronized via NTP (accuracy ±10ms); Logical counter: increments the logical value when the physical time is the same or delayed (e.g., T = (NTP time, logical value)). Global ordering: When comparing events across nodes, prioritize sorting by physical time. If they are the same, sort by logical value. Event time correction: Window recalculation is performed at the storage layer for late data (e.g., data that arrives later than the processing time due to network latency) (Apache Flink’s Allowed Lateness mechanism).

[0019] The system uses an adaptive compression algorithm, including: using the Gorilla compression algorithm for time series data to delete redundant timestamp bytes of continuous data; using the Zstandard algorithm for log text data, dynamically adjusting the compression level (1-22), and selecting the balance point between compression speed and compression ratio based on CPU load.

[0020] The federated coordinator implementation at the control layer: The master coordinator is elected through the Raft protocol and is responsible for the persistence and synchronization of cluster metadata (node ​​status, shard mapping table). When a node is overloaded, the optimal scaling strategy is predicted based on the Q-learning algorithm, and the start and stop of the edge layer collection agent are automatically scheduled.

[0021] Working steps: Collect node indicators at the kernel layer through eBPF probes and dynamically adjust the sampling frequency according to CPU / memory load; perform data preprocessing at the edge layer, including sliding window aggregation, rule engine filtering and HLC timestamp injection; distribute data to real-time channels or batch channels according to priority and transmit them to multi-level storage engines; detect real-time abnormal events based on the streaming computing framework and generate root cause reports in combination with offline analysis; monitor system status through the federated coordinator and dynamically adjust sharding strategies and resource allocation.

[0022] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A distributed data status acquisition system, characterized in that: Includes the following level modules: Edge layer: Deployed on distributed nodes, it includes a lightweight collection agent based on eBPF technology, which is used to non-invasively collect system metrics and perform adaptive sampling and local data preprocessing. Transport layer: supports dual-mode transmission protocols, including real-time channels (gRPC + Protocol Buffers) and batch channels (Kafka + Avro), and hierarchical transmission based on data priority; Storage layer: uses a multi-level storage engine, including time series database (hot storage), column storage (warm storage), and object storage (cold storage), and dynamically shards based on business tags; Computing layer: Integrates streaming computing framework and batch processing engine to support real-time event processing and offline data analysis; Control layer: Implements federated coordination based on the Raft protocol, dynamically manages node registration, load balancing, and scaling.

2. A distributed data status acquisition system according to claim 1, characterized in that: The edge layer further includes: an embedded rule engine (based on a WASM module) for performing pre-filtering before data is uploaded. The filtering rules include ignoring indicators with specified tags or removing duplicate logs; a local cache module that aggregates raw data using a sliding time window to generate mean, quantile, or standard deviation statistics.

3. A distributed data status acquisition system according to claim 1, characterized in that: The priority queue implementation method of the transport layer is as follows: data is divided into three levels: key events, performance indicators and regular logs. Key events (such as service downtime and error rate exceeding the threshold) take priority in occupying real-time channel bandwidth; and the weighted round-robin algorithm (WRR) is used to allocate transmission resources of batch channels.

4. A distributed data status acquisition system according to claim 1, characterized in that: The dynamic sharding strategy of the storage layer includes: hash sharding based on business tags (region, service, env) to ensure that data of the same business dimension is physically stored adjacently; automatically triggering rebalancing based on shard load monitoring to migrate hot shards to low-load storage nodes.

5. A distributed data status acquisition system according to claim 1, characterized in that: The system also includes a hybrid clock synchronization module that deploys a hybrid logical clock (HLC) at the edge layer nodes and combines the physical clock and the logical counter to generate a global ordered timestamp; During the calibration phase, the reference time is aligned using the NTP protocol, with an error range of less than 1 millisecond.

6. A distributed data status acquisition system according to claim 1, characterized in that: The system uses an adaptive compression algorithm, including: using the Gorilla compression algorithm for time series data to delete redundant timestamp bytes of continuous data; using the Zstandard algorithm for log text data to dynamically adjust the compression level (1-22 levels) and select the balance point between compression speed and compression rate based on CPU load.

7. A distributed data status acquisition system according to claim 1, characterized in that: The federated coordinator of the control layer implements the following: The master coordinator node is elected through the Raft protocol and is responsible for the persistence and synchronization of cluster metadata (node ​​status, shard mapping table); when a node is overloaded, the optimal scaling strategy is predicted based on the Q-learning algorithm, and the start and stop of the edge layer collection agent are automatically scheduled.