A method and system for fine-grained monitoring of compute cluster jobs

Abstract

The application discloses a kind of fine-grained monitoring method and system for computing cluster job.The method is: deploying a monitoring agent on each execution node of computing cluster, and mounting eBPF probe in the kernel of execution node;Cluster scheduler divides the total job of cluster to obtain multiple jobs and assigns to each execution node;Monitoring agent constructs the direct mapping of JobID-kernel event;Based on direct mapping, kernel event and its associated kernel process are fused to generate job-level observability entity;JobID-node-PID set dynamic mapping relationship is established by using probe to capture job start and stop event;Job-level observability entity is based on dynamic mapping relationship to aggregate all observable data streams of job on execution node;From the total observable data stream of monitoring target, set index data is collected;The collected data is persisted to time series warehouse, and the semantic features of set topic data are extracted as index.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing technology and relates to a fine-grained monitoring method and system for computing cluster operations. Background Technology

[0002] Computing clusters are the primary venues for data processing in scientific research. HPC typically refers to the collaborative solution of a single complex problem using a large number of execution nodes, characterized by computationally intensive operations, long execution cycles, and frequent inter-node communication; while HTC focuses on completing a massive number of independent tasks over a longer time span, characterized by high concurrency, shorter execution times, and extremely simplified inter-task communication.

[0003] Given the exponential growth in scale and complexity of high-performance computing (HPC) and high-throughput computing (HTC) clusters, existing monitoring technologies are no longer sufficient to meet the actual needs of production environments. In terms of monitoring architecture, current technologies generally adopt a node-level aggregation monitoring paradigm, which involves periodically polling the node's operating system interface (` / proc`, ` / sys` virtual file systems) through a data acquisition agent (such as Prometheus Node Exporter, Ganglia gmond) to obtain an overview of the physical server's overall resource usage (such as total CPU utilization and remaining memory). This approach abstracts the monitored object into a static resource pool, making it impossible to collect massive amounts of job process data. Since a single node in a cluster may run multiple jobs, or a cluster job may run concurrently on multiple nodes, node-level monitoring cannot establish a mapping relationship between "resource consumption → specific job," leading to difficulties in tracking monitored targets, inaccurate performance bottleneck location, and coarse-grained resource billing.

[0004] In terms of monitoring data transmission modes, existing technologies mainly adopt periodic reporting mechanisms based on pull or push. The pull model involves the central server periodically requesting data from nodes, while the push model involves nodes actively reporting at fixed intervals. Both are limited by static sampling periods, making it impossible to dynamically adjust the collection frequency according to the job lifecycle, and they also lack a reverse path for issuing control commands from the visualization layer to edge nodes (the control layer and data layer are separated).

[0005] In addition, existing monitoring solutions are usually composed of multiple independent components (such as metric collection agents, log collectors, scheduler plugins and backend storage systems), which leads to complex deployment and configuration, difficulty in maintaining version compatibility, lengthy troubleshooting chains, high overall maintenance costs, and difficulty in achieving rapid deployment and elastic scaling in a dynamic scheduling environment.

[0006] In summary, existing technologies are insufficient for providing fine-grained monitoring of massive operations across large-scale computing clusters in terms of monitoring granularity (node ​​level vs. job level), timeliness (minute level vs. second level), monitoring dimensions (basic indicators vs. fine-grained indicators), system overhead (high interference vs. low loss), and architectural complexity (heterogeneous assembly vs. integrated architecture).

[0007] Existing technologies mainly employ a multi-tool collaborative polling architecture, with representative implementations including: (1) MPCDF hpcmd centralized sampling scheme The hpcmd (HPC Monitoring Daemon) deployed at the Max Planck Computing and Data Facility (MPCDF) in Germany is implemented in Python. It periodically executes standard Linux commands (perf, ps) at fixed 4-minute (240-second) intervals, reads the / proc and / sys virtual file systems, and calls proprietary tools such as nvidia-smi and opainfo to collect node-level and job-level metrics. The collected data is forwarded to the central Splunk platform via rsyslog for storage and visualization, and users view job performance reports through a web interface. This solution employs a "job registration-periodic sampling" model. After a job starts, it registers a monitoring entry in the scheduler database, and the daemon polls and collects data at fixed time windows.

[0008] (2) Multi-dimensional indicator distributed data collection architecture Existing technologies employ heterogeneous toolchains for parallel data collection across different resource dimensions: CPU cache and floating-point operation metrics are read from hardware performance counters using likwid-perfctr; memory usage is assessed by parsing the VmRSS and VmSize fields in / proc / [pid] / status; disk I / O is collected using the rchar and wchar counters in / proc / [pid] / io or the iotop tool; network connection status is determined by traversing the / proc / [pid] / fd / directory to parse socket connections or by calling the ss command; and GPU utilization is obtained through the NVML library (nvidia-smi) or the DCGM (Data Center GPU Manager) interface. Each tool independently maintains its sampling clock, and data is output in text key-value pairs, binary structures, or structured log formats, respectively.

[0009] (3) Back-end storage solution The MPCDF solution deeply integrates Splunk as the sole data backend, with data format conforming to Splunk index specifications. Users view results through a predefined dashboard, and raw data export requires application through the management interface. Traditional cluster monitoring frameworks such as Ganglia use RRDtool (Round-Robin Database) to store time-series data. Node lists and metric types are statically defined in configuration files, and node online / offline status requires manual modification of the gmetad (Ganglia Meta Daemon) configuration, which does not support dynamic tag injection (such as job IDs or user names). In addition, the ELK (Elasticsearch-Logstash-Kibana) stack is widely used in many monitoring scenarios. It uses Logstash filters to solidify field mappings and limits the output target to Elasticsearch indexes.

[0010] (4) Data Acquisition Service Implementation Scheme The Prometheus Node Exporter parses process status by traversing the ` / proc` directory, executing the `open()`, `read()`, and `close()` system call sequence for each process to obtain statistics. When process-level monitoring is enabled, it recursively traverses the ` / proc / [pid] / fd / ` directory to count file descriptors and network connections. The Slurm cluster's Exporter polls for job resource consumption by calling the Slurm REST API or the `sstat` command, with a collection cycle typically of 30-60 seconds. Metrics are exposed via HTTP for the Prometheus server to pull (Pull mode). In high-concurrency scenarios, this approach requires maintaining an independent collection context for each job, with time-series data stored by job ID and process ID.

[0011] The disadvantages of existing technologies are as follows: 1. A fundamental lack of real-time performance and completeness. Existing solutions (such as MPCDF's 4-minute cycle) cannot capture performance fluctuations of jobs within a second-level time window, and lack atomic tracking of critical lifecycle events such as process fork and child process exit, resulting in incomplete data collection. Users cannot diagnose instantaneous bottlenecks, and system administrators find it difficult to analyze the true resource utilization.

[0012] 2. Severe fragmentation of multi-dimensional data. Existing solutions require multiple independent tools, such as likwid-perfctr, / proc / [pid] / io, and nvidia-smi, to collect performance data for different dimensions such as CPU, memory, I / O, and network. The sampling clocks of these tools are not synchronized, making data correlation difficult. This results in high maintenance costs and poor scalability, and makes it impossible to comprehensively distinguish between CPU-intensive, memory-intensive, I / O-intensive, or network-intensive bottlenecks.

[0013] 3. Closed ecosystem and rigid output format. Existing solutions (such as MPCDF being forced to bind to Splunk and Windows HPC being bound to ELK) have fixed output formats and cannot flexibly adapt to diverse needs such as Prometheus time-series metrics, Elasticsearch JSON documents, Kafka event streams, or local JSON / CSV, and can only use a single monitoring stack.

[0014] 4. Uncontrollable performance overhead Existing solutions require frequent parsing of multiple / proc virtual files and multiple system calls. Traversing process fd directories to count network connections has a time complexity of O(n×m). When monitoring 200+ concurrent jobs on a single node, the number of collection threads and memory usage increase dramatically, making it impossible to balance high-frequency collection (e.g., 100ms) with system overhead. Summary of the Invention

[0015] To address the problems existing in the prior art, the present invention aims to provide a fine-grained monitoring method and system for computing cluster jobs. This invention proposes a full-link observability method for computing clusters, covering edge acquisition, transmission buffering, processing storage, and visualization feedback. Through a collaborative mechanism of fine-grained metric sampling and key event capture, it achieves low-overhead, fine-grained monitoring of the entire job execution process. The invention employs an adaptive sampling algorithm and on-demand injected eBPF probes to perform high-frequency and accurate acquisition of job processes across dimensions such as CPU cycles, memory allocation, and I / O latency, while keeping the average monitoring overhead within 3% of the job's computational resource consumption. This systematically achieves fine-grained monitoring of massive jobs across high-throughput computing clusters.

[0016] The technical solution of this invention is as follows: A fine-grained monitoring method for computing cluster jobs includes the following steps: Deploy a monitoring agent on each execution node of the computing cluster and mount an eBPF probe in the kernel of each execution node; The cluster scheduler divides all jobs in the cluster into multiple jobs and assigns them to various execution nodes; The monitoring agent associates kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping between "JobID and kernel event". Based on this direct mapping, it merges kernel events and their associated kernel processes to generate job-level observable entities. It uses eBPF probes to capture job start and stop events and extracts the process tree information of the executing job, establishing a dynamic mapping relationship between "JobID, node, and PID set". The job-level observable entity aggregates all observable data streams of the job on the execution node based on this dynamic mapping relationship; PID stands for process identifier. The selected task is used as the monitoring target, and the set indicator data is collected from all observable data streams of the monitoring target. The collected data is persisted to a time-series warehouse, and semantic features of the data for a specific topic are extracted from the collected data as an index.

[0017] Preferably, the thread pool model aggregated by collection type binds a corresponding worker thread to each collector instance on the execution node and maintains a collection job list; when data collection is performed, the monitoring indicators of the current job are dynamically connected to the collection job list of the corresponding collector instance in a hot-swappable manner; when it is detected that the collection delay of a process exceeds the period threshold or the job list length exceeds the limit, the collection job list of the collector instance corresponding to the process is automatically split based on consistent hashing to achieve horizontal scaling of the collector instance of the process.

[0018] Preferably, high-frequency events are aggregated locally using eBPF kernel-mode pre-aggregation technology, and when the batch threshold is reached or the sampling period expires, they are pushed in batches through a queue; and the batch request-response mode of Netlink sockets is used to obtain the operating system kernel's statistical information on IO.

[0019] Preferably, the collected data is delivered to a distributed message queue via a double-buffered asynchronous architecture. The double-buffered asynchronous architecture is used to separate the data acquisition buffer and the backend write buffer. After the collector instance writes the collected data to the frontend data acquisition buffer, it immediately returns to continue data acquisition. The writer thread obtains the data to be written through buffer exchange, realizing the complete asynchronous nature of the acquisition path and the I / O path. An exponential backoff retry strategy is adopted, which configures the set key indicator data to retry infinitely to ensure eventual consistency, and discards non-key indicator data after N failed retry acquisition attempts.

[0020] Preferably, the distributed message queue implements a dual-path routing strategy: the collected full amount of raw indicator data is persisted to a time-series data warehouse; and the set topic data is diverted to a stream processing engine to extract semantic features as indexes.

[0021] Preferably, the eBPF probe captures the dynamic mapping relationship of the "JobID-node-PID set" maintained in memory, and assigns unlabeled kernel events to the corresponding jobs in real time; the dynamic mapping relationship of the "JobID-node-PID set" is pre-filled by the user-space controller according to the job start event of the cluster scheduler, and becomes invalid when the job terminates.

[0022] Preferably, the collected data includes, but is not limited to: monitoring single file I / O statistics, I / O request stack latency of parallel file systems, process stack tracing, CPU scheduling latency and context switching, memory allocation / page fault anomalies, high-frequency system call time consumption, and TCP retransmission / packet loss events.

[0023] Preferably, the collected data is divided into three levels—hot, warm, and cold—based on time frequency for hierarchical storage and downsampling management. Hot data is stored in memory or high-speed SSD to retain the original sampling accuracy; warm data is stored on a standard disk and downsampled using the average or maximum value; cold data is stored in object storage, retaining only statistical summaries and abnormal event records.

[0024] Preferably, the visualization front-end queries historical data and renders multi-dimensional views, and issues active collection and control commands to the monitoring agent, forming a closed-loop monitoring system from perception, transmission, analysis to feedback.

[0025] A fine-grained monitoring system for computing cluster jobs is characterized by comprising a cluster scheduler, a storage module and multiple execution nodes, with a monitoring agent deployed on each execution node and an eBPF probe mounted in the kernel of each execution node. The cluster scheduler is used to divide the full cluster job into multiple jobs and allocate them to each execution node. The monitoring agent is used to associate kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping between "JobID-kernel event"; and to fuse kernel events and their associated kernel processes based on the direct mapping to generate job-level observable entities; to capture job start and stop events using eBPF probes and extract process tree information of the processes executing the job from them, establishing a dynamic mapping relationship between "JobID-node-PID set"; the job-level observable entity aggregates all observable data streams of the job on the execution node based on the dynamic mapping relationship; PID is the process identifier; and to collect set indicator data from all observable data streams of the monitored job as the monitoring target. The storage module is used to persist the collected data to a time-series warehouse and extract the semantic features of the set topic data from the collected data as an index.

[0026] The advantages of this invention are as follows: 1) Rapid deployment, atomized scaling Employing a single-file, plug-in architecture, it enables one-click batch deployment across hundreds or thousands of execution nodes. When expanding new data collection capabilities, it can support the unified collection of job-level metrics, parallel event tracing (Trace), and heterogeneous resource data simply by hot-loading the plug-in, without replacing the main program or interrupting running jobs.

[0027] 2) Multi-dimensional observation at the operational level It integrates Slurm and HTCondor environments to enable automatic injection of job-level metadata (job ID, user, queue, resource limits). Through a pluggable pipeline, it achieves one-time instrumentation and multi-dimensional collaborative data collection: synchronously capturing resource consumption (CPU / memory / IO), communication library behavior (MPI / OpenMP), and kernel events (scheduling latency) throughout the job lifecycle, building a fine-grained full-link profile for each job, greatly reducing the interference of Probe Effect caused by multi-head collection.

[0028] 3) Low resource consumption and light CPU load To address the stringent resource exclusivity requirements of scientific computing tasks, a streaming direct-deployment architecture is adopted. Collected data is not persisted to disk; after peak shaving by a configurable memory queue, it is directly pushed to backend storage. The local stateless design completely entrusts the fault domain to the backend cluster. Performance metrics only collect raw values, with delayed aggregation and computation to the backend. The eBPF probe employs CO-RE (CompileOnce–Run Everywhere) technology, requiring only one privileged mode for kernel adaptation during initial loading; subsequent runs do not require continuous privilege escalation. Event streams are initially aggregated locally before being uploaded, ensuring that data collection has virtually no impact on the CPU usage of computing tasks, and memory usage is kept below 50MB, minimizing performance pollution for critical jobs. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 This is a diagram of the platform architecture design.

[0031] Figure 3 This is a diagram of the edge acquisition system architecture.

[0032] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0033] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0034] like Figure 1 As shown, an optional embodiment of the present invention provides a fine-grained monitoring method for computing cluster jobs, the steps of which include: Deploy a monitoring agent on each execution node of the computing cluster and mount an eBPF probe in the kernel of each execution node; The cluster scheduler divides all jobs in the cluster into multiple jobs and assigns them to various execution nodes; The monitoring agent associates kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping between "JobID and kernel event". Based on this direct mapping, it merges kernel events and their associated kernel processes to generate job-level observable entities. It uses eBPF probes to capture job start and stop events and extracts the process tree information of the executing job, establishing a dynamic mapping relationship between "JobID, node, and PID set". The job-level observable entity aggregates all observable data streams of the job on the execution node based on this dynamic mapping relationship; PID stands for process identifier. The selected task is used as the monitoring target, and the set indicator data is collected from all observable data streams of the monitoring target. The collected data is persisted to a time-series warehouse, and semantic features of the data for a specific topic are extracted from the collected data as an index.

[0035] One embodiment of the present invention proposes a fine-grained monitoring method for cluster job kernel mode based on eBPF: eBPF technology is used to mount non-intrusive probes in the kernel for real-time kernel-mode monitoring of all jobs in the cluster. Kernel events are semantically associated with job identifiers (JobIDs) through the cgroup_id / PID namespace, establishing a direct "JobID-kernel event" mapping for fine-grained job-level monitoring. The "JobID-kernel event" mapping refers to directly associating JobIDs and kernel events through JobID-PID and PID-kernel event, where PID refers to the process ID in the execution node.

[0036] The JobID is a unique identifier for a job-level observable entity in this invention, corresponding to a globally unique ID assigned to a job by the cluster scheduling system (such as HTCondor or Slurm). The association mechanism between kernel events and JobIDs is as follows: each kernel event captured by the eBPF probe carries the cgroup_id and PID namespace ID of the initiating process. By querying the "cgroup_id / PID namespace → JobID" mapping table maintained in memory, unlabeled kernel events are assigned to the corresponding jobs in real time. This mapping table is pre-populated by the user-space controller based on the job start event of the cluster scheduler and becomes invalid when the job terminates.

[0037] Fine-grained metrics collected through eBPF include, but are not limited to: monitoring single file I / O statistics, I / O request stack latency of parallel file systems, process stack tracing, CPU scheduling latency and context switching, memory allocation / page fault exceptions, high-frequency system call time, TCP retransmission / packet loss events, etc.

[0038] The BPF Map is used to aggregate high-frequency events in the kernel space (histogram, counter), waiting for user-space programs to directly query the BPF Map. Compared with the traditional / proc polling mechanism, it achieves high-precision and low-latency data acquisition, breaks through the bottleneck of coarse granularity and high overhead of traditional monitoring, and provides a data foundation for job-level fine-grained monitoring.

[0039] One embodiment of the present invention proposes a method for monitoring the entire lifecycle of a job at the job level based on the semantic association of JobID in a computing cluster: Based on the "JobID-kernel event" direct mapping constructed in the first aspect, this invention constructs a Job-level Observability Entity (JOE)—a logical object proposed in this invention, generated through the fusion processing of cluster scheduler events and kernel process tracing information. JOE uses JobID as a unique identifier and aggregates all observable data streams of a job on a single node.

[0040] A job-level observable entity, uniquely identified by JobID, is constructed to automatically capture job start and stop events and extract process tree information, establishing a dynamic mapping relationship of "JobID-node-PID set." This dynamic mapping relationship refers to which execution nodes each job is assigned to (a single JobID may run on multiple execution nodes) and which processes within those execution nodes execute it. This dynamic mapping relationship is used by the job-level observable entity to aggregate all observable data streams of the job on the execution nodes. Then, when a job is selected as a monitoring target, set indicator data is collected from all observable data streams of that monitoring target. A pre-collection and backtracking mechanism is introduced. A circular buffer is activated at the initial stage of job startup to record collected indicators and events. When the job is confirmed as a monitoring target through a rule engine or manual triggering, the historical data in the buffer is merged into the formal data stream. Unconfirmed jobs are automatically reclaimed after the buffer expires.

[0041] One embodiment of the present invention proposes a plug-in collector management method for heterogeneous event access and dynamic load balancing: The architecture employs a dual-mode triggering system. Manual triggering mode, executed via control commands (add_job / remove_job), is suitable for manual intervention and emergency diagnosis. Automatic triggering mode captures kernel scheduling events through eBPF probes, and a Lua-based rule engine matches and filters jobs, triggering automatic registration only when a preset set of rules is met. A thread pool model aggregates data by collection type (e.g., CPU / memory, I / O, network), with each collector instance bound to a dedicated worker thread and maintaining a job list. When a new job is triggered, it is dynamically added to the running collector instance's job list via hot-plugging, avoiding re-initialization. An adaptive load balancing mechanism is also built-in; when a single-threaded collection latency exceeds a periodic threshold or the job list length exceeds the limit, consistent hash-based splitting is automatically triggered to achieve horizontal scaling of collector instances.

[0042] One embodiment of this invention proposes an adaptive fine-grained acquisition and kernel-user space co-optimization method: An adaptive sampling algorithm is employed to automatically reduce the sampling frequency under high workload and resume high-frequency sampling under low workload. Through eBPF kernel-level pre-aggregation technology, histograms, counters, and other aggregation structures are maintained in the BPF Map to locally aggregate high-frequency events such as CPU cycles, memory allocation, and I / O latency. When the batch threshold is reached or the sampling period expires, batch pushing is performed through a queue. A batch request-response mode using Netlink sockets is adopted to replace traditional / proc filesystem I / O to obtain operating system kernel statistics on I / O. Global state lazy loading is implemented, with configuration parameters only atomically updated when a change event is triggered; cached copies are used during regular sampling periods to avoid lock contention.

[0043] One embodiment of the present invention proposes a method for adapting dual-buffered asynchronous delivery and pluggable storage: A logically decoupled dual-buffer architecture is constructed, separating the data acquisition buffer from the backend write buffer. After the collector instance writes the collected data to the frontend data acquisition buffer, it immediately returns to continue data acquisition. The independent writer thread obtains the data to be written through buffer exchange, realizing complete asynchronization of the acquisition path and I / O path. An exponential backoff retry strategy is adopted, configuring infinite retries for critical indicators to ensure eventual consistency, while non-critical indicators are discarded after N retries. A standard write interface is defined through the Writer Interface abstraction layer, and each storage backend implements the interface independently as a plug-in, managing its own buffer state, retry strategy, and backpressure-aware flow control, dynamically adjusting the write rate to prevent backend overload.

[0044] One embodiment of the present invention proposes a method for integrated batch and stream feature engineering and multi-level time-series data storage: A dual-routing strategy is implemented in the distributed message queue: all raw metrics are consumed in parallel by consumer groups and persisted to Elasticsearch to build a time-series data warehouse; data for specific topics is routed to a stream processing engine for real-time feature engineering, extracting high-level semantic features such as resource bottleneck patterns, anomaly detection markers, and performance degradation trends based on a sliding time window, and writing them back to an independent Elasticsearch index as a semantic enhancement layer. A three-tiered hot-warm-cold storage architecture is established: hot data (current time window) is stored in memory or high-speed SSD to retain the original sampling accuracy; warm data (recent history) is stored on standard disks and downsampled using average or maximum values; cold data (archived history) is stored in object storage, retaining only statistical summaries and anomaly event records. The system automatically routes query requests to the appropriate storage tier through time-range-based partitioned indexes.

[0045] 1. Overall Architecture like Figure 2As shown, this technical solution constructs a comprehensive observability methodology covering acquisition, transmission, processing, and visualization. A JobLens monitoring agent is deployed on each execution node of the computing cluster. This agent resides in memory as a system daemon and maintains a dynamic mapping relationship between "job-node-process set" in real time.

[0046] The acquisition module within the agent initially operates in a dormant state with zero resource consumption. When the eBPF probe (i.e., a non-intrusive probe) captures a job startup event, and the rule engine precisely matches the job ID with the process set under the execution node where the job resides, if the acquisition conditions are met, or if an external manual trigger command is received via a Unix Domain Socket, the collector is immediately dynamically instantiated and enters the active state. After activation, high-performance event probes are mounted based on eBPF to collect fine-grained metrics for computational jobs; for example, monitoring single file I / O statistics and I / O request stack latency of parallel file systems (such as Lustre / EOS) to accurately locate computational bottlenecks. The collected data is aggregated locally in kernel space via a BPF Map. During the data processing phase, the collected raw metrics are serialized according to the protocol and pushed to a distributed message queue to achieve asynchronous decoupling and traffic shaping. Then, a dual-path routing strategy is implemented: all data is persisted to Elasticsearch to build a highly available time-series data warehouse, while data for specific topics is pulled by the stream processing engine for real-time feature engineering, extracting deep semantic features related to the job dimension and writing them back to an independent index.

[0047] Ultimately, the visualization front-end achieves monitoring and control by calling the RESTful API of the back-end service: on the one hand, it uses the downlink control channel to issue instructions to the agent of specific nodes to achieve active collection and scheduling of target process sets; on the other hand, it uses the uplink query channel to pull multi-dimensional historical data for time-series rendering, thus completing the closed-loop management process of computing job performance data from accurate perception and efficient transmission to in-depth analysis and intuitive presentation.

[0048] 2. Acquisition Layer Method 2.1 Heterogeneous Event Access and Adaptive Filtering Method This solution supports both manual and automatic triggering modes for job access to adapt to different scenarios: Manual Trigger Mode: The system exposes a Unix Socket interface as an external control channel. External systems can trigger the instantiation of a specific data collector and dynamically add or remove data collection jobs by sending structured control commands (such as JSON-formatted add_job or remove_job requests) to this Socket. This mode is suitable for scenarios requiring manual intervention, batch maintenance, or emergency diagnostics.

[0049] Automatic Trigger Mode: The system attaches an eBPF (Extended Berkeley Packet Filter) probe to kernel job scheduling-related probe points (such as sys_exit_execve) to achieve kernel-space interception of job startup events. The system provides a user-space rule engine implemented in Lua scripts. The captured event data is processed by the user-space rule engine, which can match and filter job attributes (including but not limited to user ID, job name, command-line arguments, environment variables, etc.). The job is automatically registered in the collector only when its attributes meet the preset rule set. This solution achieves non-intrusive automated monitoring coverage.

[0050] 2.2 Dynamic Sharding and Load Balancing Method Based on Job Characteristics like Figure 3 As shown, when triggering data collection for a job, in order to achieve lightweight management of collection resources and load balancing in high-concurrency scenarios, this solution adopts a thread pool allocation strategy that aggregates by collection type.

[0051] Collectors perform logical aggregation based on their collection dimension type (such as CPU / memory, I / O, network, etc.), and each type of collector instance is bound to a dedicated worker thread. Each collector instance internally maintains a collection job list to manage the set of monitored jobs that are currently active.

[0052] When a new computing job triggers a data collection request, the system first queries the current load status of the data collector instance of that type. If the data collector instance is already in the running state, the job's metadata (JobID, PID set, data collection parameters, etc.) is hot-plugged into the existing data collection job list of the data collector instance without instantiating a new data collector instance, thus significantly reducing the overhead of data collector instance creation and destruction.

[0053] The system has a built-in adaptive load balancing mechanism. When it detects that the single-round collection time of a collection thread exceeds the preset collection cycle threshold (i.e., a scheduling delay occurs), or the length of the job list exceeds the configured limit, it automatically triggers horizontal scaling of the collector instance corresponding to the collection thread, instantiating a second collector shard of the same type; or it performs consistent hashing on the existing job list of the collector instance corresponding to the collection thread, migrating some jobs to the new instance, ensuring that the single-threaded load is always kept within an acceptable range.

[0054] 2.3 Spatiotemporal data compensation methods for short jobs To address the high proportion of short-lived jobs and short job durations (such as minute-level computational tasks in high-energy physics) in high-throughput computing (HTC) environments, this solution introduces pre-collection and backfill mechanisms: In the initial stage of job startup, regardless of whether the jobs in the job set currently meet the filtering conditions of the rule engine or whether a manual addition instruction has been received, the system activates the pre-collection mode by default. At this time, the collector instance writes performance metrics and eBPF events to a ring buffer or a memory queue defined by a time window at a preset sampling frequency (e.g., 1 second). The buffer duration is configurable (default is half a minute).

[0055] When the job is subsequently confirmed as a valid monitoring target (through rule matching or manual triggering), the system performs a job registration confirmation operation, atomically merging the historical data in the buffer into the formal data stream. This ensures that performance data is not lost during the initial job startup phase, achieving data integrity. If the job is not confirmed as a monitoring target within the buffer, the buffer reclamation mechanism is triggered, automatically discarding temporary data to free up memory resources.

[0056] This pre-acquisition mechanism supports differentiated sampling strategies: during the pre-acquisition phase, frequency reduction sampling (e.g., 5-second intervals) can be used to reduce system overhead, and after the job is confirmed, it can switch to high-frequency acquisition (e.g., 1-second intervals) to achieve a dynamic balance between data integrity and system load.

[0057] 2.4 Kernel-User Space Collaborative Data Acquisition and Pre-aggregation Method When collecting indicator data, this solution implements multi-level optimizations to the collection logic to address system call overhead and data transmission efficiency in high-frequency collection scenarios: 1) Kernel-mode data acquisition mechanism optimization For metrics that traditional solutions require frequent traversal of the / proc virtual file system (such as / proc / stat, / proc / net / dev, etc.) to obtain, this solution uses the Netlink Socket mechanism to replace standard file I / O operations. By establishing an asynchronous communication channel with the kernel, multi-dimensional statistical information is obtained at once in a batch request-response mode, significantly reducing the frequency of open() / read() / close() system calls, thereby reducing the overhead of user-mode / kernel-mode context switching.

[0058] 2) Global state delayed update strategy For global configuration parameters and system state variables that are relied upon during the data collection process, a lazy loading mechanism combined with cache invalidation is adopted. Atomic updates to global variables are only performed when a configuration change event is triggered or a preset cache expiration timestamp is reached. During regular data collection cycles, cache copies are directly referenced, avoiding lock contention and memory barrier overhead in each collection iteration, thus improving the execution efficiency of the collection path.

[0059] 3) eBPF data pre-aggregation optimization When using eBPF technology to collect fine-grained performance events (such as CPU cycles, memory allocation, I / O latency, etc.), in-kernel aggregation can be implemented within the eBPF program. By maintaining an aggregation counter and histogram structure in the BPF map, high-frequency raw events are aggregated in-place before being pushed to user space. Aggregation results are pushed in batches only when the batch threshold is reached or the sampling period expires. This method greatly reduces the frequency of data copying and system calls to the Ring Buffer, reducing the communication bandwidth pressure between kernel space and user space.

[0060] 2.5 Backpressure Sensing-Based Reliable Delivery Method for Streaming Data To achieve high throughput, low latency, and fault-tolerant data persistence, this invention employs a double-buffering asynchronous write architecture: 1) Logical decoupling and asynchronous pipelines The data acquisition buffer (Collection Buffer) is logically decoupled from the backend write buffer (Flush Buffer). The collector returns immediately after writing data to the frontend data acquisition buffer, without blocking and waiting for I / O to complete; the independent writer thread obtains the data to be written through the buffer swap mechanism, realizing full asynchronization between the acquisition path and the I / O path, and avoiding the backpressure from disk I / O or network latency from being transmitted to the acquisition layer.

[0061] 2) Reliable delivery based on retry mechanism The writer employs at-least-once delivery semantics and features a built-in automatic retry mechanism. When a single write operation fails due to network jitter or temporary unavailability of backend services, the system automatically retains the data in a buffer and retries up to N times (2 recommended) based on an exponential backoff strategy. Data dropping is only performed and an error log is logged when the number of retries has been exhausted and the failure is confirmed to be non-transient. For critical metrics, an infinite retry mode can be configured to ensure eventual consistency of high-value monitoring data.

[0062] 3) Plug-in backend adaptation The dual-buffered architecture decouples itself from specific storage backends through an abstract writer interface, supporting pluggable integration with various storage backends such as Prometheus, Elasticsearch, Kafka, and Time Series Databases (TSDB). Each backend writer independently manages its own buffer state and retry strategy, dynamically adjusting the write rate through a backpressure-aware flow control mechanism to prevent sudden traffic surges from overloading the backend services.

[0063] 3. Transmission and Buffering Layer Methods The collected raw metrics data, after being serialized using a protocol (such as JSON), are first delivered to a distributed message queue (such as Apache Kafka). This message queue acts as a high-throughput buffer layer, enabling asynchronous decoupling and load leveling between the data collection and storage ends.

[0064] The raw data stream is routed using a dual-routing strategy in Kafka: 1) Full data path: All raw data is consumed in parallel by consumer groups and persisted to the Elasticsearch cluster to build a searchable time-series data warehouse.

[0065] 2) Feature Extraction Path: For specific data topics with analytical value, real-time feature engineering is performed via a stream processing engine (such as Apache Spark Streaming or Flink) to extract high-level semantic features (such as resource bottleneck patterns, anomaly detection markers, performance trend indicators, etc.). This feature data serves as a semantic enhancement layer for the original indicators and is written back to an independent index in Elasticsearch, supporting complex queries and aggregation analysis.

[0066] 4. Processing and Storage Layer Methods 4.1 Feature Engineering Methods Integrating Streaming and Batch Processing (Spark / Flink Processing) To address the issue that raw metric data only reflects instantaneous states and lacks business semantics, this method deploys a stream processing engine (such as Apache Flink or Spark Streaming) in the feature extraction path to perform real-time feature engineering on topic-specific data consumed from a distributed message queue. The engine extracts high-level semantic features from the raw performance metrics based on sliding time windows or event-time semantics, including but not limited to resource bottleneck pattern recognition, anomaly detection markers, performance degradation trend indicators, and multi-dimensional resource association features. These feature data serve as a semantic enhancement layer for the raw metrics and are written back to an independent index in the time-series data warehouse. This constructs a feature dataset that supports complex aggregation queries and machine learning analysis, achieving unification of real-time stream processing and offline batch processing at the storage layer.

[0067] 4.2 Multi-level Time Series Data Indexing and Downsampling Method To address the trade-off between cost and query efficiency in storing massive amounts of time-series data, this method establishes a lifecycle management strategy based on data popularity: performance data is divided into three storage tiers according to the collection timestamp: hot data (within the current time window, stored in memory or high-speed SSD), warm data (recent history, stored on standard disks), and cold data (archived data, stored in object storage). Differentiated downsampling strategies are implemented for different tiers—hot data retains the original sampling accuracy, warm data is downsampled using the average or maximum value to reduce storage density, and cold data only retains statistical summaries and abnormal event records. Simultaneously, a time-range partitioned indexing mechanism is established, enabling query requests to be automatically routed to the corresponding storage tier based on the time dimension. This ensures low latency for real-time monitoring while achieving long-term, low-cost storage of historical data.

[0068] 5. Visualization and Feedback Control Layer Methods The visual frontend interacts with the backend service by calling a RESTful API, providing two-way control over the monitored content: Downlink control channel: Provides a graphical interface to trigger active data collection, and issues job registration instructions to the JobLens agent of a specific worker node by calling the Unix DomainSocket interface; Upstream query channel: Pulls historical performance data of jobs from Elasticsearch and renders the data through time-series visualization components to present multi-dimensional visualization views including but not limited to resource utilization time-series graphs, heatmaps, and topology relationship graphs, to assist users in performance diagnosis and resource optimization decisions.

[0069] like Figure 4 As shown, an optional embodiment of the present invention provides a fine-grained monitoring system for computing cluster jobs, characterized in that it includes a cluster scheduler, a storage module and multiple execution nodes, a monitoring agent is deployed on each execution node, and an eBPF probe is mounted in the kernel of each execution node. The cluster scheduler is used to divide the full cluster job into multiple jobs and allocate them to each execution node. The monitoring agent is used to associate kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping between "JobID-kernel event"; and to fuse kernel events and their associated kernel processes based on the direct mapping to generate job-level observable entities; to capture job start and stop events using eBPF probes and extract process tree information of the processes executing the job from them, establishing a dynamic mapping relationship between "JobID-node-PID set"; the job-level observable entity aggregates all observable data streams of the job on the execution node based on the dynamic mapping relationship; PID is the process identifier; and to collect set indicator data from all observable data streams of the monitored job as the monitoring target. The storage module is used to persist the collected data to a time-series warehouse and extract the semantic features of the set topic data from the collected data as an index.

[0070] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A fine-grained monitoring method for computing cluster jobs, comprising the following steps: Deploy a monitoring agent on each execution node of the computing cluster and mount an eBPF probe in the kernel of each execution node; The cluster scheduler divides all jobs in the cluster into multiple jobs and assigns them to various execution nodes; The monitoring agent associates kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping between "JobID and kernel event". Based on this direct mapping, it merges kernel events and their associated kernel processes to generate job-level observable entities. It uses eBPF probes to capture job start and stop events and extracts the process tree information of the executing job, establishing a dynamic mapping relationship between "JobID and node-PID set". The job-level observable entity aggregates all observable data streams of the job on the execution node based on this dynamic mapping relationship. PID stands for Process Identifier; The selected task is used as the monitoring target, and the set indicator data is collected from all observable data streams of the monitoring target. The collected data is persisted to a time-series warehouse, and semantic features of the data for a specific topic are extracted from the collected data as an index.

2. The method according to claim 1, characterized in that, The thread pool model, which aggregates by collection type, binds a corresponding worker thread to each collector instance on each execution node and maintains a collection job list. When data collection is performed, the monitoring indicators of the current job are dynamically connected to the collection job list of the corresponding collector instance in a hot-swappable manner. When it is detected that the collection delay of a process exceeds the period threshold or the job list length exceeds the limit, the collection job list of the collector instance corresponding to the process is automatically split based on consistent hashing to achieve horizontal scaling of the collector instance of the process.

3. The method according to claim 2, characterized in that, High-frequency events are aggregated locally using eBPF kernel-mode pre-aggregation technology. When the batch threshold is reached or the sampling period expires, the events are pushed in batches through a queue. Additionally, Netlink sockets are used to obtain the operating system kernel's statistical information on I / O using a batch request-response model.

4. The method according to claim 2, characterized in that, The collected data is delivered to a distributed message queue via a double-buffered asynchronous architecture. The double-buffered asynchronous architecture is used to separate the data acquisition buffer and the backend write buffer. After the collector instance writes the collected data to the front-end data acquisition buffer, it immediately returns to continue data acquisition. The writer thread obtains the data to be written through buffer exchange, realizing the complete asynchronous nature of the acquisition path and the I / O path. An exponential backoff retry strategy is adopted, which configures the set key indicator data to retry indefinitely to ensure eventual consistency, and discards non-key indicator data after N failed retry acquisition attempts.

5. The method according to claim 4, characterized in that, The distributed message queue implements a dual-path routing strategy: all collected raw indicator data is persisted to a time-series data warehouse; and topic-specific data is routed to a stream processing engine to extract semantic features as indexes.

6. The method according to claim 1, 2, or 3, characterized in that, eBPF probe captures unlabeled kernel events by querying the dynamic mapping relationship of the "JobID-node-PID set" maintained in memory, and assigns them to the corresponding jobs in real time. The dynamic mapping relationship of "JobID-node-PID set" is pre-populated by the user-space controller based on the job start event of the cluster scheduler and becomes invalid when the job terminates.

7. The method according to claim 1, characterized in that, The collected data includes, but is not limited to: monitoring single file I / O statistics, I / O request stack latency of parallel file systems, process stack tracing, CPU scheduling latency and context switching, memory allocation / page fault exceptions, high-frequency system call time consumption, and TCP retransmission / packet loss events.

8. The method according to claim 1, characterized in that, Based on the time frequency, the collected data is divided into three levels: hot, warm, and cold, and then stored and downsampled. Hot data is stored in memory or high-speed SSD to retain the original sampling accuracy; warm data is stored on standard disks and downsampled using the average or maximum value; cold data is stored in object storage, retaining only statistical summaries and records of abnormal events.

9. The method according to claim 1, characterized in that, The visualization front-end queries historical data and renders multi-dimensional views, as well as sends active collection and control commands to the monitoring agent, forming a closed-loop monitoring system from perception, transmission, analysis to feedback.

10. A fine-grained monitoring system for computing cluster operations, characterized in that, It includes a cluster scheduler, a storage module, and multiple execution nodes. A monitoring agent is deployed on each execution node, and an eBPF probe is mounted in the kernel of each execution node. The cluster scheduler is used to divide the full cluster job into multiple jobs and allocate them to each execution node. The monitoring agent is used to associate kernel events of execution nodes with job identifiers (JobIDs) to construct a direct mapping of "JobID-kernel event"; and to fuse kernel events and their associated kernel processes based on the direct mapping to generate job-level observable entities; to capture job start and stop events using eBPF probes and extract process tree information of the processes executing the job from them, establishing a dynamic mapping relationship of "JobID-node-PID set"; the job-level observable entity aggregates all observable data streams of the job on the execution node based on the dynamic mapping relationship; PID stands for Process Identifier; And to use the user-selected job as the monitoring target, and collect set indicator data from all observable data streams of the monitoring target; The storage module is used to persist the collected data to a time-series warehouse and extract the semantic features of the set topic data from the collected data as an index.

Images