Data sampling method and device based on multi-dimensional load adaptation

Abstract

The application provides a data sampling method and device based on multi-dimensional load adaptation, the method comprising: acquiring system load data, target process load data and collector load data, and calculating a comprehensive load factor; storing captured multi-dimensional events into a per-CPU ring event stack in a kernel state, performing time sequence alignment on events of the same tracking identifier based on a time wheel time sequence index structure to generate composite data; and dynamically adjusting a time wheel granularity and a maximum waiting window according to the comprehensive load factor, and reporting the composite data to a user state for data sampling. The application can reduce inherent scheduling delay and data copying overhead associated with the user state, achieve cross-CPU event association while ensuring write performance, achieve dynamic balance between alignment accuracy and processing efficiency, and achieve adaptive change of kernel state processing overhead with system load.

Description

Technical Field

[0001] This application belongs to the field of distributed tracing, and in particular relates to a data sampling and device based on multidimensional load adaptive tracing. Background Technology

[0002] In the observability domain of cloud-native distributed systems, distributed tracing requires correlating multi-dimensional events of the same request to support fault localization. In existing technologies, the correlation operations for multi-dimensional events are generally performed in user space, with kernel-level eBPF tools only responsible for event collection and basic filtering. This architecture is limited by user-space scheduling latency and cross-kernel user-space data transfer overhead, failing to meet the requirements for microsecond-level correlation. Furthermore, existing load adaptation schemes only adjust the event sampling rate and reporting frequency, without addressing the dynamic adjustment of kernel-level event processing logic. When system load increases, complex kernel-level processing may further exacerbate resource contention. The time wheel mechanism is typically used as a periodic task scheduler in eBPF environments, and its capabilities as a time-series index structure are not effectively utilized. Summary of the Invention

[0003] The purpose of this invention is to provide a data sampling and device based on multi-dimensional load adaptation, so as to realize microsecond-level time-series correlation of multi-dimensional events in cloud-native distributed tracing scenarios, while controlling the overall overhead of the observation system within a preset safety range.

[0004] This application discloses a data sampling method based on multidimensional load adaptation, including: Acquire system load data, target process load data, and collector load data, and calculate the comprehensive load factor; In kernel mode, captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU, and the time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window. The time wheel granularity and maximum waiting window are dynamically adjusted based on the comprehensive load factor, and the composite data is reported to the user space for data sampling.

[0005] Optionally, acquire system load data, target process load data, and collector load data, and calculate a comprehensive load factor, including: The system layer's CPU utilization and memory utilization are collected as the system load data; the target process's CPU utilization and thread pool saturation are collected as the target process load data; and the current backlog and event loss rate of the event buffer are read as the collector load data. The system load data, target process load data, and collector load data are normalized respectively, and the comprehensive load factor is obtained by weighted summation according to preset weights.

[0006] Optionally, in kernel mode, captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data, including: For any event in any dimension, extract the event's tracking identifier and dimension type, and store the event in the per-CPU circular event stack corresponding to the current CPU; Use the tracking identifier as the key to query the global causal index table, update or create the causal context, where the global causal index table is a hash table that stores the mapping relationship between tracking identifiers and corresponding event location metadata; The causal context is inserted into the corresponding slot of the time wheel time sequence index structure according to the event timestamp. When all the events of all dimensions corresponding to the causal context have arrived or exceeded the maximum waiting window, the corresponding events are aggregated to generate composite data.

[0007] Optionally, dynamically adjusting the time wheel granularity and maximum wait window based on the comprehensive load factor includes: When the comprehensive load factor is less than the first threshold, the time wheel granularity is set to the first granularity, and the maximum waiting window is set to the first window. When the comprehensive load factor is greater than or equal to the first threshold and less than or equal to the second threshold, the time wheel granularity is linearly adjusted to between the first granularity and the second granularity, and the maximum waiting window is linearly adjusted to between the first window and the second window. When the comprehensive load factor is greater than the second threshold, the time wheel granularity is set to the second granularity, and the maximum waiting window is set to the second window, wherein the second granularity is greater than the first granularity, and the second window is greater than the first window.

[0008] Optionally, reporting the composite data to user space includes: The composite data is divided into different levels according to its priority. Write composite data of different levels into the corresponding per-CPU cache queue; The composite data in each per-CPU cache queue is reported to user space in batches according to the time interval corresponding to the level.

[0009] Optionally, after inserting the causal context into the corresponding slot of the time wheel sequence index structure based on the event timestamp, the method further includes: Set a dynamic timeout based on the current slot's event count and historical event arrival rate; When the event count of any slot exceeds the event density threshold or there is an urgent pending context, the time wheel sequence index structure is immediately advanced, the causal context linked list of the current slot is traversed, and the alignment operation is performed.

[0010] Optional, also includes: When an event with an emergency flag is captured, query the corresponding causal context; The causal context is removed from the linked list of the slot in the original time wheel sequence index structure and inserted into the head of the linked list of the current slot, triggering the time wheel sequence index structure to advance immediately and prioritize the processing of the causal context.

[0011] Optionally, aggregate corresponding events to generate composite data, including: Obtain the CPU number and slot index corresponding to each dimension of the event from the causal context; Read event data from the per-CPU circular event stack of the corresponding CPU based on the CPU number and slot index; Sort all event data by timestamp to generate composite data.

[0012] Optionally, before the kernel stores the captured multi-dimensional events into the per-CPU circular event stack, the following steps are also included: The sampling rate for the corresponding dimension is determined based on the comprehensive load factor. The captured events are sampled based on the sampling rate; The events that pass sampling are stored in the per-CPU circular event stack.

[0013] This application also discloses a data sampling device based on multidimensional load adaptation, comprising: The load calculation module is used to acquire system load data, target process load data, and collector load data, and calculate the comprehensive load factor. The kernel-mode alignment module is used to store captured multi-dimensional events into a per-CPU circular event stack. Based on the time wheel timing index structure, it performs timing alignment on events with the same tracking identifier to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU, and the time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window. The parameter adjustment module is used to dynamically adjust the time wheel granularity and the maximum waiting window based on the comprehensive load factor. The output module is used to report the composite data to the user space.

[0014] As can be seen from the above technical solution, this application calculates the comprehensive load factor by synchronously acquiring load data from three dimensions: system, target process, and collector. This allows multi-dimensional event association operations to be executed in kernel mode, reducing the inherent scheduling latency and data copy overhead of user-mode association. Furthermore, it employs a per-CPU circular event stack to store event data, achieving lock-free writing. Combined with a global causal index table that only stores metadata, it achieves cross-CPU event association while ensuring write performance. Simultaneously, it transforms the traditional time wheel into a time-series index structure to store the context to be aligned. The time wheel is driven by event density and urgent events, achieving a dynamic balance between alignment accuracy and processing efficiency. In addition, the time wheel granularity and maximum waiting window are dynamically adjusted based on the comprehensive load factor, allowing kernel-mode processing overhead to adaptively change with system load, avoiding negative impacts of observation behavior on business operations. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of an embodiment of the data sampling method based on multidimensional load adaptation provided in this application; Figure 2 A flowchart for calculating the comprehensive load factor by acquiring system load data, target process load data, and collector load data, as provided in this application; Figure 3 The flowchart provided in this application shows how to store captured multi-dimensional events into a per-CPU circular event stack in kernel mode, and how to perform time-series alignment of events with the same tracking identifier to generate composite data based on the time wheel timing index structure. Figure 4 The flowchart provided in this application illustrates the process of reporting the composite data to user space. Figure 5 A flowchart of another embodiment of the data sampling method based on multidimensional load adaptation provided in this application; Figure 6 A flowchart of yet another embodiment of the data sampling method based on multidimensional load adaptation provided in this application; Figure 7 A flowchart for generating composite data for the aggregated corresponding events provided in this application; Figure 8 The structural block diagram of the data sampling device based on multidimensional load adaptation provided in this application is shown. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] This application provides a data sampling method based on multidimensional load adaptation, applied to the observability data acquisition and processing scenario of a cloud-native distributed system (hereinafter referred to as the system).

[0019] As distributed systems grow in scale and business complexity increases, system observability becomes increasingly important. Distributed tracing, a crucial component of observability, helps developers and operations personnel trace the propagation path of requests within a distributed system, pinpointing system faults and performance bottlenecks. To achieve distributed tracing, it's necessary to collect multi-dimensional events generated by the same request across different components and layers, including network events, system call events, application log events, and database access events. These events are then linked together according to chronological order and causal relationships to form a complete request call chain.

[0020] The inventors discovered through research that existing distributed tracking technologies have, but are not limited to, the following technical problems: Problem 1: Multi-dimensional event correlation is generally performed in user space, with the kernel only responsible for event collection. This architecture requires transferring a large number of raw events from kernel space to user space, and cross-space data transfer and user space scheduling introduce significant latency and overhead. To improve event write performance, some solutions use per-CPU queues to store events. However, per-CPU queues cannot directly achieve global event correlation; aggregation and sorting are required in user space, further increasing processing latency. Other solutions use global queues to store events in kernel space. While this facilitates global correlation, global queues suffer from severe lock contention during multi-core concurrent access, resulting in poor write performance and an inability to handle high-concurrency scenarios.

[0021] Question 2: Regarding the use of the time wheel, existing technologies typically use it as a periodic task scheduler to execute tasks on a timed basis. Each time slot stores a list of tasks to be executed, and the time wheel advances according to a fixed period. This fixed-period advancement method generates a large number of empty advances when the event density is low, wasting CPU resources; when the event density is high, the fixed advancement period cannot process the accumulated events in a timely manner, leading to increased latency.

[0022] Question 3: Regarding load adaptive adjustment, existing load adaptive solutions typically only adjust the event sampling rate and reporting frequency, without addressing the dynamic adjustment of kernel-level event processing logic. When system load increases, even if the sampling rate is reduced, the overhead of kernel-level event processing may still impact the business system.

[0023] To address at least one of the aforementioned technical problems, this disclosure provides a data sampling method based on multidimensional load adaptation, such as... Figure 1 As shown, it includes the following steps: S100 acquires system load data, target process load data, and collector load data, and calculates the comprehensive load factor.

[0024] To ensure that kernel-mode event processing does not negatively impact the normal operation of the business system, this application introduces a multi-dimensional load adaptive adjustment mechanism. Specifically, the system collects load data in real time from three dimensions: the system, the target business process, and the collector itself, and calculates a comprehensive load factor that comprehensively reflects the overall operating status of the system.

[0025] System load data reflects the resource usage of the entire operating system and is an important basis for judging the overall health of the system.

[0026] The target process load data reflects the running status of the observed business processes. The normal operation of business processes is the primary goal of the system, so the load status of business processes should be given priority.

[0027] The data from the data collector reflects the operating status of the observation system itself. The processing capacity of the data collector directly affects the quality of event acquisition and processing. If the data collector is overloaded, it may lead to event loss or increased processing delay.

[0028] By simultaneously collecting load data from these three dimensions, we can comprehensively and accurately grasp the overall operating status of the system, providing a data foundation for subsequent comprehensive load assessment and parameter adjustment.

[0029] S200: In kernel mode, the captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU. The time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window.

[0030] In this embodiment of the application, by sinking the associated operations of multi-dimensional events to the kernel mode for execution, and combining them with a multi-dimensional load adaptive adjustment mechanism, low-latency and low-overhead event timing alignment is achieved.

[0031] In existing technologies, multi-dimensional event correlation in distributed tracing systems is generally executed in user space, while kernel-space eBPF tools are only responsible for event collection and basic filtering. This architecture has significant limitations. First, raw events collected in kernel space need to be sent to user space via data transmission channels. Cross-kernel-user space data transmission incurs additional CPU and memory overhead, especially in high-concurrency scenarios where the transmission of a large number of raw events consumes significant system resources. Second, the scheduling of user-space processes is affected by the operating system scheduler, resulting in unpredictable scheduling delays. When the system load is high, the user-space correlation process may fail to process arriving events in a timely manner, leading to increased correlation latency and even event backlog and loss. Finally, user-space correlation requires the aggregation and sorting of events from different CPUs and probes, a process that consumes substantial computing resources, further exacerbating the system load.

[0032] In this embodiment, the correlation operations of multi-dimensional events are moved from user space to kernel space for execution. Timing alignment is completed at the source of the event, generating composite data containing complete correlation information before being reported to user space. This setup avoids the transmission of a large number of raw events between user space and kernel space, reducing data transmission overhead. Simultaneously, the execution in kernel space is unaffected by user space scheduling, ensuring the real-time performance of correlation operations and reducing correlation latency.

[0033] Specifically, to achieve efficient event storage and association in kernel mode, this application employs a per-CPU circular event stack to store event data. The per-CPU design ensures that each CPU core has an independent event storage area. Once an event is generated, it can be directly stored in the current CPU's stack without requiring synchronization or locking operations between multiple cores, significantly improving event write performance. The circular event stack uses a pre-allocated, fixed-size array structure, eliminating the need for dynamic memory allocation in kernel mode and avoiding the overhead and risks associated with dynamic memory allocation. Furthermore, the circular structure naturally supports cyclic event overwriting; when the stack is full, it automatically overwrites the oldest event, preventing memory overflow.

[0034] To achieve multi-dimensional event association under the same trace identifier (hereinafter represented by trace_id), this application constructs a time-wheel time-series index structure (hereinafter referred to as the time wheel). Unlike traditional time wheels, the time wheel in this application is not used to schedule timed tasks, but rather serves as a time-series index structure. Each time slot corresponds to a time window, storing pointers to all causal contexts to be aligned within that window. A causal context is a data structure used to record the location information of events in various dimensions under the same trace_id, using trace_id as a unique identifier. When a new event arrives, the system searches for the corresponding causal context based on the event's trace_id. If it does not exist, a new context is created, the event's location information is recorded in the context, and the context pointer is inserted into the slot of the corresponding time window of the time wheel. When the time wheel advances to a slot, the system iterates through all causal contexts in that slot, checking whether events in all dimensions have arrived or whether the maximum waiting window has been exceeded. If the conditions are met, the events in the corresponding dimensions are aggregated to generate composite data. This time-wheel-based sequential index structure enables centralized management of events within the same time window, facilitating batch processing and timeout judgment, and improving the efficiency of associated operations.

[0035] S300: Based on the comprehensive load factor, dynamically adjust the time wheel granularity and the maximum waiting window, and report the composite data to the user space for data sampling.

[0036] In this embodiment, the granularity of the time wheel and the maximum waiting window are dynamically adjusted based on changes in the comprehensive load factor. When the system load is low, a smaller time wheel granularity and a shorter maximum waiting window are used to ensure the accuracy and real-time performance of event alignment. When the system load increases, the time wheel granularity is automatically increased and the maximum waiting window is extended to reduce the frequency of time wheel advancement and the overhead of event processing. This avoids excessive resource consumption by the observation system, which could affect the operation of the business system. This adaptive adjustment mechanism enables the observation system to dynamically adjust its behavior according to the actual operating status of the system, ensuring alignment accuracy while keeping system overhead within a preset safety range.

[0037] Compared to related technologies, this application calculates a comprehensive load factor by synchronously acquiring load data from three dimensions: the system, the target process, and the collector. This decentralizes multi-dimensional event correlation operations to kernel mode, eliminating the inherent scheduling latency and data copying overhead of user-mode correlation. A per-CPU circular event stack is used to store event data for lock-free writing. Combined with a global causal index table that only stores metadata, cross-CPU event correlation is achieved while ensuring write performance. The traditional time wheel is transformed into a time-series index structure to store the context to be aligned. The time wheel is driven by event density and urgent events, achieving a dynamic balance between alignment accuracy and processing efficiency. The granularity of the time wheel and the maximum waiting window are dynamically adjusted based on the comprehensive load factor, allowing kernel-mode processing overhead to adaptively change with system load and avoiding negative impacts of observation behavior on business operations.

[0038] Optional, such as Figure 2 As shown, system load data, target process load data, and collector load data are acquired, and a comprehensive load factor is calculated, including: S110, collect the CPU utilization and memory utilization of the system layer as the system load data, collect the CPU utilization and thread pool saturation of the target process as the target process load data, and read the current backlog and event loss rate of the event buffer as the collector load data.

[0039] In this embodiment, system load data mainly includes CPU utilization and memory utilization. The CPU is one of the most important resources in a computer system, and CPU utilization directly reflects the busyness of CPU resources, making it the most commonly used indicator for judging system load. Memory utilization reflects the usage of system memory resources. When memory utilization is too high, the system may experience insufficient memory, leading to process termination or slow system operation. These two indicators reflect the overall load status of the system from different perspectives, are representative, and are easy to collect. For example, CPU utilization can be calculated by reading the operating system's process statistics file. In a Linux kernel-based system, the `stat` file in the `proc` directory can be read. This file records statistical information such as user-mode time, kernel-mode time, idle time, and I / O wait time for all CPU cores since system startup. By reading this file twice consecutively and calculating the time difference between the two reads and the changes in time for each state, the CPU utilization per unit time can be obtained. Memory utilization can be calculated by reading the `meminfo` file in the `proc` directory. This file records information such as the total system memory, free memory, buffer memory, and cache memory. By calculating the ratio of used memory to total memory, the memory utilization rate can be obtained.

[0040] It should be noted that system load data may also include other indicators that reflect the system's operating status, such as average system load, context switch count, interrupt rate, disk I / O utilization, and network bandwidth utilization. Average system load refers to the average length of the run queue over a period of time, reflecting the overall load pressure on the system. Context switch count refers to the number of times the CPU switches between different processes per unit of time; excessively frequent context switching can lead to decreased system performance. Interrupt rate refers to the number of hardware and software interrupts processed by the system per unit of time; a high interrupt rate can also affect system performance. Disk I / O utilization refers to the proportion of time the disk is in a busy state per unit of time, reflecting the utilization of disk resources. Network bandwidth utilization refers to the ratio of the amount of data transmitted through the network interface per unit of time to the total bandwidth, reflecting the utilization of network resources. Those skilled in the art will understand that appropriate system load indicators can be selected according to the needs of the actual application scenario; this application is not limited to these.

[0041] In this embodiment, the target process load data mainly includes the target process's CPU utilization and thread pool saturation. The target process refers to the observed business process, such as a web service process, database service process, or application service process. The target process's CPU utilization refers to the proportion of CPU time used by the target process per unit of time, reflecting the target process's consumption of CPU resources. Thread pool saturation refers to the ratio of the number of running threads in the target process to the total number of threads in the thread pool, reflecting the target process's thread resource usage. When the thread pool saturation is close to 1, it indicates that the target process's thread resources have been exhausted, and new requests need to wait for thread release, leading to increased request processing latency. These two indicators directly reflect the running status and processing capacity of the target business process and are important bases for judging whether the business process is running normally. For example, the target process's CPU utilization can be calculated by reading the target process's statistics file. In a Linux kernel-based system, the stat file corresponding to the process ID in the proc directory can be read. This file records statistical information such as user-mode time and kernel-mode time since the target process started. By reading this file twice consecutively and calculating the time difference between the two reads and the change in the target process's CPU time, the target process's CPU utilization can be obtained. Thread pool saturation can be calculated by reading the thread state information of the target process. In Linux kernel-based systems, the total number of threads in the target process can be obtained by reading the number of subdirectories under the task directory corresponding to the process ID in the proc directory. Then, by reading the state information of each thread, the number of threads in the running state can be counted, thereby calculating the thread pool saturation.

[0042] It should be noted that the target process load data may also include other indicators that reflect the running status of the target process, such as the target process's memory utilization, virtual memory size, number of open file descriptors, request queue length, average response time, and requests per second. The target process's memory utilization refers to the ratio of the memory used by the target process to the total system memory, reflecting the target process's consumption of memory resources. The number of open file descriptors refers to the number of files, sockets, and other resources currently open by the target process. When the number of file descriptors reaches the system limit, the target process will be unable to open new resources, leading to service anomalies. The request queue length refers to the number of requests waiting to be processed in the target process; an excessively long request queue will increase request processing latency. The average response time is the average time it takes for the target process to process requests, reflecting the target process's processing performance. Requests per second refers to the number of requests processed by the target process per second, reflecting the target process's throughput capacity. Those skilled in the art will understand that appropriate target process load indicators can be selected according to the needs of the actual application scenario; this application is not limited to these.

[0043] In this embodiment, the collector's load data mainly includes the current backlog of the event buffer and the event loss rate. A collector refers to a system used to collect and process observable data. In this application, the collector includes a kernel-mode eBPF probe and a user-mode handler. The event buffer is used to transfer event data between kernel mode and user mode. When the kernel-mode probe captures an event, it writes the event to the event buffer. The user-mode handler reads the event from the event buffer and performs subsequent processing. The current backlog of the event buffer refers to the number of events written to the event buffer but not yet read by user mode, reflecting the degree of matching between the collector's processing capacity and the event generation rate. When the event generation rate exceeds the user-mode processing rate, the backlog of the event buffer will continuously increase, eventually leading to buffer overflow and event loss. The event loss rate is the ratio of the number of events lost per unit time to the total number of events, reflecting the collector's data loss situation. These two indicators directly reflect the collector's own operating status and processing capacity, and are important bases for judging whether the collector can work normally. For example, the current backlog of the event buffer can be obtained by reading the event buffer's status information. When using eBPFRingBuffer as the event buffer, the used space and total space of the RingBuffer can be queried using the corresponding query function to calculate the current backlog. The event loss rate can be calculated by counting the number of events dropped by the kernel and the total number of events. When the kernel probe writes to the event buffer, if it finds that the buffer is full, it drops the event and counts it, and then periodically reports the count information to the user space. The user space calculates the event loss rate based on the total number of events and the number of dropped events.

[0044] It should be noted that the collector load data may also include other indicators that reflect the collector's operating status, such as the collector process's CPU utilization, memory utilization, eBPF Map utilization, user-mode consumption latency, and event processing latency. The collector process's CPU and memory utilization reflect the collector's own consumption of system resources. eBPF Map utilization refers to the ratio of used entries to the total number of entries in the eBPF Map. When Map utilization is too high, new entries cannot be inserted, resulting in data loss. User-mode consumption latency refers to the time interval from when an event is written to the event buffer to when it is read by the user, reflecting the response speed of the user-mode processor. Event processing latency refers to the time interval from when an event is captured to when processing is completed, reflecting the overall processing performance of the collector. Those skilled in the art will understand that appropriate collector load indicators can be selected according to the needs of the actual application scenario; this application is not limited to this.

[0045] S120, the system load data, target process load data and collector load data are normalized respectively, and the comprehensive load factor is obtained by weighted summation according to preset weights.

[0046] In this embodiment, the system load data, target process load data, and collector load data are normalized to eliminate differences in dimensions and value ranges between different indicators, making them comparable and enabling weighted summation. Different load indicators have different dimensions and value ranges; for example, CPU utilization ranges from 0 to 100, memory utilization also ranges from 0 to 100, while event buffer backlog ranges from 0 to the maximum buffer capacity, and thread pool saturation ranges from 0 to 1. If these indicators are directly weighted and summed, indicators with larger value ranges will have an excessive impact on the overall load factor, causing the overall load factor to fail to accurately reflect the true load status of the system. Therefore, it is necessary to first normalize all indicators to the same value range, typically between 0 and 1, before performing weighted summation.

[0047] There are many methods for normalization. In this embodiment, linear normalization is used to map the value of each indicator to a range of 0 to 1. The formula for linear normalization is: the normalized value equals the current value minus the minimum value, then divided by the maximum value minus the minimum value. For indicators whose value range is already 0 to 1 or 0 to 100, a simple conversion can be performed directly, such as dividing the percentage value by 100 to obtain a value between 0 and 1. For example, the value range of CPU utilization is 0 to 100, so the normalized CPU utilization is equal to the CPU utilization value divided by 100. The value range of memory utilization is also 0 to 100, so the normalized memory utilization is equal to the memory utilization value divided by 100. The value range of thread pool saturation is 0 to 1, so no additional normalization is required, and it can be used directly. The value range of event buffer backlog is 0 to the maximum capacity of the buffer, so the normalized event buffer backlog is equal to the current backlog divided by the maximum capacity of the buffer. If the current backlog exceeds the maximum capacity of the buffer, the normalization value is set to 1.

[0048] In this embodiment, the normalized system load factor, target process load factor, and collector load factor are weighted and summed according to preset weights to obtain the comprehensive load factor. The comprehensive load factor is calculated as follows: the comprehensive load factor equals the system load factor multiplied by α, plus the target process load factor multiplied by β, plus the collector load factor multiplied by γ, where α, β, and γ are the preset weights of the system load factor, target process load factor, and collector load factor, respectively, and the sum of α, β, and γ is 1. The weight settings need to be determined based on the importance of different dimensions of load in the actual application scenario. Generally, the running status of the target business process is the most important to ensure, so the weight β of the target process load factor is usually set to the highest; the overall system load status is also important, so the weight α of the system load factor is next; the load status of the collector itself is relatively less important, so the weight γ of the collector load factor is next. For example, α can be set to 0.2, β to 0.6, and γ to 0.2, so that the target process load factor has the greatest impact on the comprehensive load factor, while the system load factor and collector load factor have relatively smaller impacts. The reason for this configuration is that the primary goal of the observability system is to support the business system and not affect its normal operation. Therefore, the load status of the business process needs to be considered first. Secondly, the overall load status of the system will also affect the operation of the business process, so it also needs to be taken into account. Finally, the load status of the collector itself reflects the processing capacity of the collection system and also needs to be considered appropriately.

[0049] Those skilled in the art will understand that the values ​​of the three weights, α, β, and γ, can be adjusted according to the needs of the actual application scenario. For example, in scenarios with high requirements for collector performance, the load status of the collector has a significant impact on the system, so the weight of γ can be appropriately increased; in scenarios with high requirements for overall system stability, the load status of the system has a significant impact on the business, so the weight of α can be appropriately increased; in scenarios with extremely high requirements for business process performance, the load status of the business process is the most important factor to consider, so the weight of β can be further increased. As long as the sum of the three weights is 1, this application is not limited to this.

[0050] It should be noted that the calculation method for the comprehensive load factor is not limited to linear weighted summation; other methods can also be used, such as the maximum value method, minimum value method, geometric mean method, and exponential weighted method. The maximum value method takes the maximum value among the system load factor, target process load factor, and collector load factor as the comprehensive load factor. This method reflects the most severe load situation in the system but ignores the load situation in other dimensions. The minimum value method takes the minimum value among the three factors as the comprehensive load factor. This method is too conservative and cannot reflect the true load situation of the system. The geometric mean method multiplies the three factors and takes the cube root to obtain the comprehensive load factor. This method is sensitive to changes in all three factors, but when one factor is 0, the comprehensive load factor is also 0, which does not reflect reality. The exponential weighted method assigns an exponential weight to each factor and then multiplies them to obtain the comprehensive load factor. This method can highlight the impact of high load factors. Those skilled in the art will understand that an appropriate comprehensive load factor calculation method can be selected according to the needs of the actual application scenario, and this application is not limited to this.

[0051] In this embodiment, storing the captured multi-dimensional events into a per-CPU circular event stack in kernel mode is a key step in achieving efficient event storage. After the eBPF probe captures events at various key points in kernel mode, the event data needs to be stored for subsequent timing alignment and correlation processing. The performance of event storage directly affects the throughput and latency of the entire acquisition system, especially in high-concurrency scenarios where millions of events may be generated per second, requiring a high-performance, low-overhead event storage mechanism.

[0052] Optional, such as Figure 3 As shown, in kernel mode, captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data, including: S210, for any event in any dimension, extract the tracking identifier and dimension type of the event, and store the event in the per-CPU circular event stack corresponding to the current CPU.

[0053] In this embodiment, the trace_id and dimension type of the captured event are first extracted. The trace_id is an identifier used in distributed tracing to uniquely identify a request; all events generated by the same request across different components and layers share the same trace_id. The trace_id allows association of all events belonging to the same request. The dimension type refers to the source and category of the event, such as network dimension, system dimension, application dimension, database dimension, metric dimension, etc. Events with different dimensions have different content and uses; the dimension type allows for categorized management and processing of events. For example, when the XDP probe captures a network packet event, the trace_id can be extracted from the packet header. Typically, the trace_id is added to the HTTP header or other protocol fields of the request. Then, the dimension type of the event is marked as network. When the syscall probe captures a system call event, the trace_id can be extracted from the process context or system call parameters, and the dimension type of the event is marked as system. When the uprobe probe captures an application log event, the trace_id can be extracted from the log content, and the dimension type of the event is marked as application.

[0054] It should be noted that if the captured event does not contain a trace_id, a temporary trace_id can be generated. The temporary trace_id can be generated based on information such as the event's timestamp, process ID, thread ID, and CPU number, ensuring its uniqueness. After generating the temporary trace_id, it can be added to the event data for subsequent association. Those skilled in the art will understand that appropriate trace_id extraction and generation methods can be selected according to the needs of the actual application scenario; this application is not limited to this.

[0055] In this design, events are stored in a per-CPU circular event stack corresponding to the current CPU. A per-CPU circular event stack is a fixed-size, pre-allocated array structure independently owned by each CPU core. Each CPU core has its own independent event stack. When an event is captured on a CPU core, it is directly stored in that CPU's event stack without accessing the event stacks of other CPUs. The advantage of this per-CPU design is that it completely avoids lock contention between multiple cores. In multi-core systems, if a global event queue is used to store events, when multiple CPU cores write events to the queue simultaneously, a locking mechanism is needed to ensure the atomicity of queue operations. Lock contention leads to a decrease in write performance, especially in high-concurrency scenarios, where lock contention can become a system bottleneck. The per-CPU design ensures that each CPU only accesses its own event stack without any synchronization operations, resulting in a significant improvement in write performance.

[0056] In practice, the circular event stack uses a pre-allocated, fixed-size array structure. The array size is determined during system initialization, eliminating the need for dynamic memory allocation in kernel mode. Dynamic memory allocation in kernel mode incurs additional overhead and may fail, impacting system stability. The pre-allocated array structure avoids these problems, ensuring the stability and predictability of event storage. Furthermore, the circular structure inherently supports cyclic event overwriting. When the stack pointer exceeds the maximum array length, it automatically wraps back to the beginning of the array, overwriting the oldest event. This prevents stack overflow and ensures the system doesn't run out of memory due to an excessive number of events. When the oldest event is overwritten, event loss information can be recorded for subsequent statistical analysis.

[0057] For example, each CPU's circular event stack can contain 4096 slots, each 256 bytes in size, used to store the data of one event. The stack pointer is initialized to 0. When a new event arrives, the event data is written to the slot pointed to by the stack pointer, and then the stack pointer is incremented by 1. If the stack pointer equals 4096, it is reset to 0, overwriting the oldest event at the beginning of the array. Each slot also contains a status flag to indicate whether the slot is idle or in use. After an event is read and processed, the corresponding slot is marked as idle so that it can be reused for subsequent events.

[0058] S220: Query the global causal index table using the tracking identifier as the key, and update or create the causal context. The global causal index table is a hash table that stores the mapping relationship between tracking identifiers and corresponding event location metadata.

[0059] The global causal index table is a hash table that stores the mapping relationship between trace_id and corresponding event location metadata. Hash tables are an efficient data structure, capable of performing lookup, insertion, and deletion operations with an average time complexity of O(1), making them ideal for mapping trace_id to causal context. A causal context is a data structure used to record the event location information for each dimension of the same trace_id. Each causal context corresponds to a unique trace_id, containing the CPU number and slot index of all dimension events for that trace_id, as well as alignment completion flags, creation timestamps, and other information.

[0060] When a new event arrives, first extract the event's `trace_id`, then query the global causal index table using that `trace_id` as the key. If the query result is empty, it means this is the first event for that `trace_id`, and a new causal context needs to be created. During the initialization of the new causal context, the CPU ID and slot index for all dimensions are set to invalid values, indicating that events for that dimension have not yet arrived. Then, the CPU ID and slot index of the current event are filled into the corresponding dimension positions, the alignment completion flag is initialized to 0, and the context creation timestamp is recorded. Finally, the newly created causal context is inserted into the global causal index table. If the query result is not empty, it means the causal context for that `trace_id` already exists, and the context content needs to be updated. The CPU ID and slot index of the current event are filled into the corresponding dimension positions, and then it is checked whether events for all dimensions have arrived. If events for all dimensions have arrived, the alignment completion flag is set to 1, indicating that the context is ready for alignment and aggregation.

[0061] Optionally, the global causal index table is protected using bpf_spin_lock to ensure data consistency during concurrent access by multiple cores. Since the global causal index table only stores location metadata of events, rather than complete event data, the size of a single record is very small, typically not exceeding 64 bytes. Furthermore, each trace_id only needs to be inserted and deleted from the global causal index table once, resulting in a relatively low access frequency. Therefore, the probability of lock contention is very low, the average lock holding time is very short, and it will not have a significant impact on system performance.

[0062] S230, the causal context is inserted into the corresponding slot of the time wheel time sequence index structure according to the event timestamp. When all the events of all dimensions corresponding to the causal context have arrived or exceeded the maximum waiting window, the corresponding events are aggregated to generate composite data.

[0063] In this embodiment, the causal context is inserted into the corresponding slot of the time wheel sequence index structure according to the event timestamp. The time wheel sequence index structure is an array structure that stores pointers of causal contexts to be aligned within the corresponding time window. The time wheel consists of a fixed number of slots, each slot corresponding to a time window, the size of which is determined by the time wheel granularity. For example, if the time wheel granularity is 100 microseconds, then each slot corresponds to a 100-microsecond time window. When a causal context is created or updated, the slot number to which the context should be inserted is calculated based on the event timestamp and the current time wheel granularity. The slot number is calculated as follows: the slot number equals the event timestamp divided by the time wheel granularity, modulo the total number of slots in the time wheel. Then, the pointer of the causal context is inserted at the head of the linked list of the corresponding slot.

[0064] Each slot in the time wheel's time-series index structure contains a context list head pointer, an event count, an earliest timestamp, and a latest timestamp. The context list head pointer points to a linked list of all causal contexts in that slot, the event count records the number of contexts in that slot, and the earliest and latest timestamps record the time range of all events in that slot. This information allows for convenient management and processing of contexts to be aligned within each time window. When the time wheel advances to a slot, it only needs to traverse the context list of that slot and process each context individually, eliminating the need to traverse all contexts and significantly improving processing efficiency.

[0065] In this embodiment, when all dimensional events corresponding to a causal context have arrived or exceeded the maximum waiting window, the corresponding events are aggregated to generate composite data. For each causal context, the system checks two conditions: one is whether the alignment completion flag is 1, meaning all dimensional events have arrived; the other is whether the current time exceeds the context's creation time plus the maximum waiting window. If either condition is met, the aggregation operation is triggered, aggregating the events of the corresponding dimension to generate composite data.

[0066] If the alignment completion flag is 1, it means that all dimension events for that trace_id have arrived, and complete aggregation can be performed. The system obtains the CPU number and slot index of each dimension event from the causal context, and then reads the event data from the corresponding CPU's circular event stack based on this information. Event data from all dimensions is collected, sorted by timestamp, and then encapsulated into a composite data structure. This composite data structure contains trace_id, event data for each dimension, the event time order, alignment completion time, and other information.

[0067] If the current time exceeds the context's creation time plus the maximum waiting window, it indicates that some dimension events for that trace_id may be lost or delayed. To prevent the context from occupying memory resources indefinitely, timeout processing is required. In this case, the system will aggregate the arrived dimension events to generate composite data, while recording the missing dimension information in the composite data. This ensures that even if some events are lost, some usable related data can still be generated, rather than completely discarding all events for that trace_id.

[0068] After the composite data is generated, it will be reported to user space for further processing and storage. At the same time, the system will release the resources occupied by the causal context, mark the corresponding slot in the event stack as free, delete the context from the global causal index table, and remove the context pointer from the linked list of the time wheel slot, so that subsequent events can reuse these resources.

[0069] In this embodiment, dynamically adjusting the time wheel granularity and maximum wait window based on the comprehensive load factor is the core step in achieving adaptive adjustment. Time wheel granularity refers to the size of the time window corresponding to each time slot, while the maximum wait window refers to the longest time the system waits for all dimension events of a given trace_id to arrive. These two parameters directly affect the accuracy of event alignment and system overhead. Smaller time wheel granularity results in higher alignment accuracy, but also a higher time wheel advancement frequency and greater CPU overhead; larger time wheel granularity results in lower alignment accuracy, but a lower time wheel advancement frequency and less CPU overhead. A shorter maximum wait window results in more timely event reporting, but may lead to more events being lost due to timeouts; a longer maximum wait window results in fewer lost events, but higher event latency and greater memory usage. Therefore, it is necessary to dynamically adjust these two parameters based on the actual system load to achieve an optimal balance between alignment accuracy and system overhead.

[0070] Optionally, dynamically adjusting the time wheel granularity and maximum wait window based on the comprehensive load factor includes: A1, when the comprehensive load factor is less than the first threshold, the time wheel granularity is set to the first granularity, and the maximum waiting window is set to the first window; A2, when the comprehensive load factor is greater than or equal to the first threshold and less than or equal to the second threshold, the time wheel granularity is linearly adjusted to between the first granularity and the second granularity, and the maximum waiting window is linearly adjusted to between the first window and the second window; A3, when the comprehensive load factor is greater than the second threshold, the time wheel granularity is set to the second granularity, and the maximum waiting window is set to the second window, wherein the second granularity is greater than the first granularity, and the second window is greater than the first window.

[0071] In this embodiment, the first threshold and the second threshold are two critical values ​​used to divide the load range, with the first threshold being less than the second threshold. The first threshold represents the upper limit of the system's low-load state. When the comprehensive load factor is less than the first threshold, it indicates that the load on the system, business processes, and the collector is low, system resources are sufficient, and the overhead of the collection system will not affect the normal operation of the business system. The second threshold represents the lower limit of the system's high-load state. When the comprehensive load factor is greater than the second threshold, it indicates that the load on the system, business processes, or the collector is already high, and the overhead of the collection system may affect the normal operation of the business system. Therefore, it is necessary to reduce the overhead of the collection system and prioritize the operation of the business system. When the comprehensive load factor is between the first and second thresholds, it indicates that the system is in a medium-load state, requiring gradual adjustment of the time wheel parameters to control the overhead of the collection system while ensuring a certain alignment accuracy.

[0072] For example, the first threshold can be set to 0.6, and the second threshold can be set to 0.8. When the comprehensive load factor is less than 0.6, the system is in a low load range; when the comprehensive load factor is between 0.6 and 0.8, the system is in a medium load range; and when the comprehensive load factor is greater than 0.8, the system is in a high load range. Those skilled in the art will understand that the specific values ​​of the first and second thresholds need to be determined based on the performance test results of the actual application scenario. Different systems have different performance characteristics and load-bearing capacities, so the threshold settings will also differ. For example, on a high-performance server, the system has a stronger load-bearing capacity, so the first and second thresholds can be set higher; on a low-performance server, the system has a weaker load-bearing capacity, so the first and second thresholds need to be set lower. Furthermore, the threshold settings can also be adjusted according to the importance of the business. For highly important businesses, to avoid the data collection system affecting the business, the threshold can be set lower, and parameter adjustments can begin in advance; for less important businesses, the threshold can be set higher to ensure alignment accuracy as much as possible. This application is not limited to this.

[0073] In this embodiment, when the overall load factor is less than a first threshold, the system is in a low-load range. At this time, system resources are sufficient, business processes are running normally, and the collector can process all event data promptly. Therefore, the time wheel granularity is set to the first granularity, and the maximum waiting window is set to the first window. The first granularity is a smaller time wheel granularity, and the first window is a shorter maximum waiting window. This ensures the accuracy and real-time performance of event alignment, minimizing event delays and loss. For example, the first granularity can be set to 100 microseconds, and the first window can be set to 500 microseconds. Thus, each time slot corresponds to a 100-microsecond time window, and the system waits a maximum of 500 microseconds before processing and reporting the context, ensuring the real-time performance of events.

[0074] When the overall load factor is greater than or equal to the first threshold and less than or equal to the second threshold, the system is in a medium load range. At this time, the system load is increasing but has not yet reached a high load state. It is necessary to gradually increase the time wheel granularity and extend the maximum wait window to reduce the frequency of time wheel advancement and event processing overhead, thus preventing the system from entering a high load state. The time wheel granularity and maximum wait window are adjusted linearly, increasing linearly with the overall load factor. This linear adjustment method enables smooth parameter changes, avoiding system jitter and data fluctuations caused by sudden parameter changes.

[0075] The formula for linear adjustment is as follows: the current time wheel granularity equals the first granularity plus the comprehensive load factor minus the first threshold, then divided by the second threshold minus the first threshold, and multiplied by the second granularity minus the first granularity. The current maximum waiting window equals the first window plus the comprehensive load factor minus the first threshold, then divided by the second threshold minus the first threshold, and multiplied by the second window minus the first window. For example, assuming the first granularity is 100 microseconds, the second granularity is 200 microseconds, the first window is 500 microseconds, and the second window is 1000 microseconds. When the comprehensive load factor is 0.7, the current time wheel granularity equals 100 plus (0.7-0.6) / (0.8-0.6) multiplied by (200-100), which equals 150 microseconds. The current maximum waiting window equals 500 plus (0.7-0.6) / (0.8-0.6) multiplied by (1000-500), which equals 750 microseconds. In this way, as the overall load factor increases from 0.6 to 0.8, the time wheel granularity increases linearly from 100 microseconds to 200 microseconds, and the maximum waiting window increases linearly from 500 microseconds to 1000 microseconds, the parameters are smoothly adjusted.

[0076] When the overall load factor exceeds the second threshold, the system is in a high-load range. At this point, the load on the system, business processes, or data collectors is already high, and the overhead of the data collection system may severely impact the normal operation of the business system. It is necessary to set the time wheel granularity to the second granularity and the maximum wait window to the second window. The second granularity is a larger time wheel granularity, and the second window is a longer maximum wait window. This can significantly reduce the frequency of time wheel advancement and the overhead of event processing, prioritizing the operation of the business system. For example, the second granularity can be set to 200 microseconds, and the second window can be set to 1000 microseconds. This halves the frequency of time wheel advancement and correspondingly reduces the overhead of event processing, effectively alleviating the system's load pressure.

[0077] It should be noted that the adjustment methods for the time wheel granularity and maximum waiting window are not limited to linear adjustment; other methods can also be used, such as stepped adjustment and exponential adjustment. Stepped adjustment involves dividing the medium load range into multiple sub-ranges, each corresponding to a fixed time wheel granularity and maximum waiting window. As the comprehensive load factor enters different sub-ranges, the parameters change stepwise. Exponential adjustment means that the time wheel granularity and maximum waiting window increase exponentially with the increase of the comprehensive load factor. This method results in slower parameter changes at lower loads and faster changes at higher loads, better handling rapid load changes. Those skilled in the art will understand that appropriate parameter adjustment methods can be selected according to the needs of the actual application scenario; this application is not limited to these methods.

[0078] In this embodiment, reporting composite data to user space is the final step in the entire event processing flow. The composite data contains event data from all arrived dimensions with the same trace_id, and has already undergone time-series alignment and correlation. Upon receiving the composite data, user space can directly store, analyze, and display it without requiring further complex correlation operations. To further reduce the processing pressure on user space and network transmission overhead, this application adopts a hierarchical, time-segmented batch reporting method, dividing the composite data into different priority levels, with different reporting strategies applied to different levels of composite data.

[0079] Optional, such as Figure 4 As shown, reporting the composite data to user space includes: S310 classifies composite data into different levels based on their priority.

[0080] The priority of composite data is determined based on the content and importance of the events it contains. Composite data with different priorities have different values ​​for system monitoring and troubleshooting, therefore different reporting strategies are required. For composite data with high importance, its real-time reporting must be ensured, and it should be reported to user space as soon as possible; for composite data with low importance, reporting can be appropriately delayed, and batch reporting can be used to reduce transmission overhead and user space processing pressure.

[0081] For example, composite data can be divided into three levels: urgent, important, and normal. Urgent-level composite data refers to composite data containing errors, anomalies, and urgent alarms. This data directly reflects system fault conditions and needs to be reported to the user space promptly so that operations and maintenance personnel can identify and handle problems in a timely manner. Important-level composite data refers to composite data containing business metrics, slow requests, and critical operations. This data reflects the system's operating status and performance and needs to ensure a certain level of real-time performance. Normal-level composite data refers to composite data of regular requests. This data is mainly used for daily system monitoring and performance analysis, and its real-time requirements are relatively low. Those skilled in the art will understand that more or fewer priority levels can be divided according to the needs of the actual application scenario; this application is not limited to this.

[0082] The S320 writes composite data of different levels to the corresponding per-CPU cache queue.

[0083] By writing composite data of different priorities into corresponding per-CPU cache queues, lock contention between multiple cores can be avoided, improving write performance. Each priority level corresponds to a per-CPU cache queue. Specifically, each CPU core has its own independent cache queue with three levels. After composite data is generated, it is written to the cache queue corresponding to the current CPU according to its priority. The per-CPU cache queue adopts a pre-allocated fixed-size array structure, eliminating the need for dynamic memory allocation and ensuring write stability and performance.

[0084] Furthermore, when the cache queue is full, new composite data will overwrite the oldest composite data in the queue. This prevents the cache queue from overflowing and avoids memory leaks. Simultaneously, data loss information is recorded for subsequent statistical analysis. For urgent composite data, to ensure its real-time performance, it can be reported directly to user space without going through the cache queue. This avoids urgent data waiting in the cache queue, ensuring timely reception by user space.

[0085] S330, according to the time interval corresponding to the level, reports the composite data in each per-CPU cache queue in batches to user space.

[0086] Different priority levels of the cache queues correspond to different reporting intervals. Urgent-level composite data does not require caching and is reported immediately; important-level composite data has a shorter reporting interval, while ordinary-level composite data has a longer reporting interval. This ensures the real-time nature of important data while reducing the reporting frequency of ordinary data, thereby reducing transmission overhead and user-space processing pressure.

[0087] For example, the reporting interval for critical composite data can be set to 1 second, and the reporting interval for ordinary composite data can be set to 5 seconds. The system uses an eBPF timer to trigger batch reporting of the cache queue. Each priority level corresponds to an independent eBPF timer, which is triggered periodically at the corresponding time interval. When the timer is triggered, the system iterates through the corresponding level's cache queues of all CPUs, writes all composite data in the queues in batches into the eBPF buffer, and reports it to user space. Then, the cache queue is cleared, and all slots are marked as free for subsequent reuse of composite data.

[0088] In this embodiment, batch reporting significantly reduces the number of context switches between user mode and kernel mode, thus lowering system overhead. Each context switch requires saving and restoring the CPU register state, which incurs overhead. If each composite data item is reported individually, it leads to numerous context switches, increasing the system's CPU overhead. Batch reporting, however, allows multiple composite data items to be reported at once, requiring only one context switch and greatly reducing the overhead of context switching. Simultaneously, batch transmission improves data transmission efficiency, reduces header overhead of the transmission protocol, and lowers network bandwidth usage.

[0089] It should be noted that the reporting interval can also be dynamically adjusted based on the system's overall load factor. When the system load increases, the reporting interval for both important and ordinary composite data can be appropriately extended to further reduce system overhead; when the system load decreases, the reporting interval should be restored to the default value. This allows for adaptive adjustment of the reporting strategy, better adapting to changes in system load. Those skilled in the art will understand that appropriate reporting intervals and dynamic adjustment strategies can be set according to the needs of actual application scenarios, and this application is not limited thereto.

[0090] In this application embodiment, the time wheel's advancement mechanism is a crucial factor affecting event alignment efficiency and system overhead. Traditional time wheels employ a fixed-cycle advancement method, advancing at a fixed interval regardless of whether there are events to process in a slot. This approach leads to numerous idle advances at low event densities, wasting CPU resources; at high event densities, the fixed advancement cycle may fail to process a large number of events in a slot in a timely manner, resulting in event backlog and increased latency. To address this issue, this application employs a hybrid advancement mechanism combining event density-driven and emergency event-triggered approaches, enabling the time wheel's advancement frequency to be dynamically adjusted based on the actual event arrival rate and emergency situation.

[0091] Optional, such as Figure 5 As shown, after inserting the causal context into the corresponding slot of the time wheel sequence index structure according to the event timestamp, it also includes: S240 sets a dynamic timeout based on the current slot's event count and historical event arrival rate.

[0092] The dynamic timeout refers to the time interval between the time wheel's advance to the next slot. It is not fixed but dynamically adjusted based on the current event density. When the event density is high, the timeout is shortened, allowing the time wheel to advance faster and process events in the slots more promptly. When the event density is low, the timeout is extended to reduce idle advances of the time wheel and lower CPU overhead.

[0093] In practice, the dynamic timeout is calculated as follows: the timeout equals the larger of the time wheel granularity and the event count multiplied by the single event processing time. The single event processing time refers to the average time required to process a causal context, which can be obtained from historical statistics. This is because when the number of events in a slot is small, the time required to process these events is short, and the timeout is mainly determined by the time wheel granularity, ensuring alignment accuracy. When the number of events in a slot is large, the time required to process these events is long, and the timeout needs to be extended accordingly to avoid excessive CPU overhead caused by overly frequent time wheel advancements.

[0094] For example, suppose the time wheel granularity is 100 microseconds and the single event processing time is 100 nanoseconds. If the event count for the current slot is 100, then the event count multiplied by the single event processing time equals 10 microseconds, which is less than the time wheel granularity of 100 microseconds. Therefore, the dynamic timeout is set to 100 microseconds. If the event count for the current slot is 2000, then the event count multiplied by the single event processing time equals 200 microseconds, which is greater than the time wheel granularity of 100 microseconds. Therefore, the dynamic timeout is also set to 200 microseconds. In this way, when the number of events is large, the time wheel advance interval will automatically increase, giving the system enough time to process these events and avoiding CPU overload.

[0095] S250: When the event count of any slot exceeds the event density threshold or there is an urgent pending context, immediately advance the time wheel sequence index structure, traverse the causal context linked list of the current slot, and perform alignment operation.

[0096] The event density threshold refers to the maximum number of contexts a single slot can hold. When the event count in a slot exceeds this threshold, it indicates that the events in that slot have accumulated to a certain level and need to be processed promptly; otherwise, it will lead to event backlog and increased latency. At this point, regardless of whether the dynamic timeout has expired, the time wheel will immediately advance to process the events in that slot. This mechanism ensures that in the event of a sudden surge, the system can respond promptly, quickly process accumulated events, and avoid excessive event latency.

[0097] Optionally, the event density threshold can also be dynamically adjusted based on the system's overall load factor. When the system load is low, the event density threshold can be set higher, allowing more events to accumulate in the slots and reducing the frequency of time wheel advancement; when the system load is high, the event density threshold can be set lower, enabling the time wheel to advance more frequently, processing events promptly, and avoiding event backlog. For example, when the overall load factor is less than the first threshold, the event density threshold can be set to 1000; when the overall load factor is between the first and second thresholds, the event density threshold can be linearly reduced to 500; when the overall load factor is greater than the second threshold, the event density threshold can be fixed at 500. This allows for adaptive adjustment of the event density threshold, better adapting to changes in system load.

[0098] In this embodiment, when an urgent pending context exists, the time wheel is immediately advanced. An urgent pending context refers to a causal context containing an urgent event; these contexts require priority processing and should be reported to user space as soon as possible. When an urgent event arrives, the system marks the corresponding causal context as urgent and immediately triggers the time wheel to advance, prioritizing the processing of that context. This mechanism ensures the real-time nature of urgent events, enabling maintenance personnel to promptly detect and handle system failures.

[0099] When the time wheel is triggered to advance, the system moves the time wheel pointer to the next slot, then traverses the causal context list of the original slot, performing an alignment operation on each context. The alignment operation includes checking if the context is ready for aggregation, i.e., all dimension events have arrived or exceeded the maximum waiting window. If the condition is met, the corresponding events are aggregated to generate composite data and reported to user space; if the condition is not met, the context is moved to the next slot, awaiting further processing. Contexts marked as urgent are prioritized, skipping the regular waiting window check, and immediately aggregated and reported.

[0100] In this embodiment, handling emergency events is crucial for ensuring the timely detection and location of system faults. In distributed systems, the occurrence of errors and abnormal events often indicates system failures that require immediate attention; otherwise, business interruptions and economic losses may result. Therefore, special handling mechanisms are needed for emergency events that include errors and abnormalities to ensure their real-time performance and priority.

[0101] Optional, such as Figure 6 As shown, the method further includes: S410, when an event with an emergency flag is captured, query the corresponding causal context.

[0102] When an event with an urgent flag is captured, the event's trace_id is first extracted. Then, using this trace_id as the key, the global causal index table is queried to obtain the corresponding causal context. If the causal context already exists, it means that other dimension events for that trace_id have arrived, and the context has already been inserted into a slot in the time wheel. At this point, the causal context needs to be removed from the linked list of the original time wheel slot and then inserted at the head of the linked list of the current time wheel slot. The purpose of this is to move the urgent context to the foremost time slot so that it can be processed with priority.

[0103] S420, the causal context is removed from the linked list of the slot in the original time wheel sequence index structure and inserted into the head of the linked list of the current slot, triggering the time wheel sequence index structure to advance immediately and prioritize the processing of the causal context.

[0104] Inserting the causal context at the head of the linked list for the current slot ensures that urgent contexts are processed first when traversing the slot linked list. The element at the head of the linked list is accessed first, so inserting the urgent context at the head ensures it is processed before other normal contexts. If the causal context does not exist, it means this is the first event for that trace_id and is an urgent event. In this case, a new causal context is created, marked as urgent, and then inserted at the head of the linked list for the current time wheel slot.

[0105] After the emergency context is inserted at the head of the linked list for the current slot, an immediate time wheel advance is triggered. Immediate time wheel advance means that the time wheel pointer is immediately moved to the current slot, and then the context linked list for that slot is traversed, prioritizing the processing of emergency contexts. This avoids emergency contexts waiting for the regular time wheel advance cycle, allowing them to be processed and reported immediately, significantly reducing the delay of emergency events.

[0106] During the time wheel's progression, contexts marked as urgent are processed first. The system first traverses all urgent contexts in the slot list, aggregates and reports them, and then processes regular contexts. For urgent contexts, the regular maximum wait window check is skipped; aggregation and reporting are performed immediately regardless of whether events for all dimensions have arrived. If events for some dimensions have not yet arrived, the missing dimension information is recorded in the composite data, and subsequent arriving events are reported to fill in the gaps. This ensures that urgent events are received by the user space as soon as possible, and even with some missing data, timely information is provided for troubleshooting.

[0107] For example, when an application crashes, the Uprobe probe captures an abnormal event, which is marked as an emergency event. The system extracts the event's trace_id, queries the global causal index table, and finds the corresponding causal context. This context was originally inserted into the 10th slot of the time wheel and would only be processed when the time wheel advances to the 10th slot. The system removes this context from the linked list of the 10th slot, inserts it at the head of the linked list of the current 5th slot, and then triggers the time wheel to advance immediately. When the time wheel pointer moves to the 5th slot, it processes the emergency context first while traversing the linked list, aggregates the arrived event data, generates composite data, and immediately reports it to user space. Upon receiving this composite data, the user space monitoring system immediately issues an alarm, notifying operations personnel to handle the fault. The entire process has a very short latency, ensuring that the fault is detected promptly.

[0108] In this embodiment, aggregating corresponding events to generate composite data is the final step in the time-series alignment process and a crucial step in integrating scattered multi-dimensional events into complete correlated data. The composite data includes event data for all reached dimensions with the same trace_id, as well as associated metadata. After receiving the composite data, the user space can use it directly without needing to perform complex association and sorting operations.

[0109] Optional, such as Figure 7 As shown, aggregated events generate composite data, including: S510 retrieves the CPU number and slot index corresponding to each dimension of the event from the causal context.

[0110] The causal context records the CPU ID and slot index for each dimension event with the same trace_id. For each dimension, if the event for that dimension has arrived, the corresponding CPU ID and slot index are valid; if the event for that dimension has not yet arrived, the corresponding CPU ID and slot index are invalid. The system iterates through all dimensions in the causal context, collecting valid CPU ID and slot index information.

[0111] S520, read event data from the per-CPU circular event stack of the corresponding CPU according to the CPU number and slot index.

[0112] In eBPF, a CPU can access not only its own ring event stack data, but also data in the ring event stacks of other CPUs. Therefore, the system can access the ring event stack of the corresponding CPU based on the CPU number and read the event data in that slot based on the slot index.

[0113] S530 sorts all event data by timestamp to generate composite data.

[0114] Since the per-CPU circular event stack is a pre-allocated fixed array structure, and the slot index directly corresponds to the array subscript, the read operation has a time complexity of O(1), which is very efficient. Moreover, once the event data is written to the event stack, the content will not be modified until it is marked as free, so no locking operation is required, and it can be read directly. This can ensure the performance and consistency of data reading.

[0115] After collecting event data across all dimensions, this data is sorted by timestamp. Each event has a kernel timestamp, recording the time the event was captured. Sorting by timestamp arranges all events with the same trace_id in chronological order, forming a complete request call chain. Sorting methods can include simple bubble sort, insertion sort, or quicksort. Since the number of dimensions for each trace_id is typically small, generally not exceeding 10, the sorting overhead is very small and will not significantly impact system performance.

[0116] Finally, the sorted event data and associated metadata are encapsulated into a composite data structure. This composite data structure includes trace_id, number of events, event data for each dimension, event time order, alignment completion time, and missing dimension information. The size of the composite data can be adjusted according to actual needs, typically not exceeding 1KB, to facilitate transmission and storage. After generation, the composite data is written to the corresponding cache queue or directly reported to user space.

[0117] It's important to note that if, during the aggregation process, event data for a certain dimension is found to have been overwritten by new events, it means that the event has remained in the event stack for too long and has been overwritten by subsequent events. In this case, the system will record the missing event information for that dimension in the composite data for subsequent statistical analysis. This situation typically occurs under extremely high system load and very high event generation rates; it is a normal phenomenon and will not affect the overall correlation effect.

[0118] In this application embodiment, adaptive sampling based on multidimensional load is an important means of controlling the overhead of the data acquisition system. In high-concurrency scenarios, millions of events may be generated per second. If all events are collected and processed, it will bring huge overhead to the system and may even affect the normal operation of the business system. Therefore, it is necessary to dynamically adjust the event sampling rate according to the actual load of the system to control the overhead of the data acquisition system while ensuring data validity.

[0119] Optionally, before the kernel stores the captured multi-dimensional events into the per-CPU circular event stack, the following steps are also included: B1, determine the sampling rate for the corresponding dimension based on the comprehensive load factor.

[0120] In this embodiment, the sampling rate for the corresponding dimension is determined based on the comprehensive load factor. The sampling rate refers to the ratio of the number of events collected to the total number of events. Events in different dimensions have different levels of importance, therefore different sampling rates are required. For dimensions with high importance, the sampling rate can be set higher to ensure that enough event data is collected; for dimensions with low importance, the sampling rate can be set lower to reduce the number of events collected and reduce system overhead.

[0121] Specifically, the sampling rate is determined based on the load range of the comprehensive load factor. When the comprehensive load factor is less than the first threshold, the system is in a low load range. At this time, system resources are sufficient, and a higher sampling rate, or even full sampling, can be used to ensure the integrity of the observed data. When the comprehensive load factor is between the first and second thresholds, the system is in a medium load range, and the sampling rate of non-critical dimensions needs to be gradually reduced to control the overhead of the acquisition system. When the comprehensive load factor is greater than the second threshold, the system is in a high load range, and the sampling rate of all dimensions needs to be further reduced to prioritize the operation of the business system.

[0122] For example, when the overall load factor is less than the first threshold, the sampling rate for all dimensions is set to 1, meaning full sampling and all events are collected. When the overall load factor is between the first and second thresholds, the sampling rate for important dimensions remains at 1, while the sampling rate for unimportant dimensions decreases linearly with the increase of the overall load factor, from 1 to 0.5. When the overall load factor is greater than the second threshold, the sampling rate for important dimensions is set to 0.8, and the sampling rate for unimportant dimensions is set to 0.2. Thus, when the system load increases, the sampling rate for unimportant dimensions decreases first, reducing the number of events collected, while the sampling rate for important dimensions remains at a high level, ensuring the collection of important events.

[0123] B2, sampling the captured events based on the sampling rate.

[0124] In this embodiment, captured events are sampled based on a sampling rate. The sampling method employs random sampling, generating a random number between 0 and 1 for each captured event. If the random number is less than or equal to the sampling rate of the current dimension, the event is sampled successfully, and subsequent processing continues; if the random number is greater than the sampling rate of the current dimension, the event is discarded, and the processing terminates. Random sampling is a simple and efficient sampling method that ensures the randomness and representativeness of the samples, making it suitable for most application scenarios.

[0125] The sampling operation is performed before the event is stored in the per-CPU circular event stack. This avoids writing unnecessary events to the event stack, saving memory space and reducing the overhead of subsequent processing. If the event is discarded, no further operations are required, and the process can simply return. If the event sampling is successful, the event is stored in the per-CPU circular event stack, and subsequent steps such as context creation and timing index insertion continue.

[0126] It should be noted that the sampling rate can also be dynamically adjusted based on the type and content of the event. For example, events containing errors and anomalies are collected regardless of the current sampling rate to ensure that these important events are not lost. For ordinary access events, sampling can be performed at the normal sampling rate. This allows for the collection of important events while reducing the sampling rate, thus improving the effectiveness of the observation data. Those skilled in the art will understand that appropriate sampling methods and sampling rate adjustment strategies can be selected according to the needs of the actual application scenario, and this application is not limited thereto.

[0127] B3 stores the sampled events into the per-CPU circular event stack.

[0128] Based on the same inventive concept, this disclosure provides a data sampling device based on multidimensional load adaptation, such as... Figure 8 As shown, it includes: The load calculation module 10 is used to acquire system load data, target process load data and collector load data, and calculate the comprehensive load factor. The kernel-mode alignment module 20 is used to store the captured multi-dimensional events into the per-CPU circular event stack, and to perform time-series alignment of events with the same tracking identifier based on the time wheel timing index structure to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU, and the time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window. Parameter adjustment module 30 is used to dynamically adjust the time wheel granularity and the maximum waiting window according to the comprehensive load factor; Output module 40 is used to report the composite data to user space.

[0129] It is understood that the specific implementation process of the load calculation module 10, kernel mode alignment module 20, parameter adjustment module 30 and output module 40 in the embodiments of this disclosure can be referred to the relevant records of the data sampling method, and will not be repeated here.

[0130] The technical solution provided in this application employs a per-CPU circular event stack to store event data, achieving lock-free high-performance writing. Simultaneously, a global causal index table storing only metadata enables cross-CPU event association, balancing write performance and association capabilities. Furthermore, the traditional time wheel is transformed into a time-series index structure, with each time slot storing a causal context pointer to be aligned. The advancement of the time wheel no longer depends on a fixed period but is driven by both event density and urgency. When event density is high, the time wheel advances quickly, processing events promptly; when event density is low, the time wheel advances slowly, saving CPU resources. Urgent events can trigger immediate time wheel advancement, ensuring the real-time nature of urgent events. In addition, a multi-dimensional load adaptive mechanism is deeply integrated with the kernel-mode time-series alignment engine. Based on the system's comprehensive load factor, multiple parameters such as time wheel granularity, maximum wait window, event density threshold, and sampling rate are dynamically adjusted, achieving adaptive adjustment of kernel-mode processing overhead. This ensures that the overhead of the observation system remains within a safe range under different load conditions, without negatively impacting the normal operation of the business system.

[0131] The following section provides a detailed explanation of the specific execution process of the data sampling method and apparatus based on multidimensional load adaptation in this application, using a complete operational example.

[0132] Assume that during system initialization, the weight α of the system load factor is set to 0.2, the weight β of the target process load factor is 0.6, the weight γ of the collector load factor is 0.2, the first threshold is 0.6, the second threshold is 0.8, the first granularity is 100 microseconds, the second granularity is 200 microseconds, the first window is 500 microseconds, the second window is 1000 microseconds, and the event density threshold is 1000. Create a per-CPU circular event stack, allocating 4096 slots per CPU. Create a global causal index table with a maximum of 1 million entries. Create a time-wheel time-series index structure with 1024 slots. Create three levels of per-CPU cache queues: urgent events report immediately, important events report every 1 second, and normal events report every 5 seconds. Attach XDP probes, syscall probes, uprobe probes, and trace point probes, and begin collecting events.

[0133] In the initial stage of system operation, the volume of business requests is low. The system's CPU utilization is 20%, memory utilization is 30%, the target process's CPU utilization is 15%, thread pool saturation is 20%, the event buffer backlog is 0%, and the event loss rate is 0%. Based on this data, the load calculation module calculates the system load factor as 0.2 x 0.7 + 0.3 x 0.3, which equals 0.23. The target process load factor is 0.15 x 0.6 + 0.2 x 0.4, which equals 0.17. The collector load factor is 0 x 0.6 + 0 x 0.4, which equals 0. The overall load factor is 0.23 x 0.2 + 0.17 x 0.6 + 0 x 0.2, which equals 0.148. The overall load factor of 0.148 is less than the first threshold of 0.6, therefore the system is in a low-load range. The parameter adjustment module sets the time wheel granularity to 100 microseconds, the maximum wait window to 500 microseconds, the event density threshold to 1000, and the sampling rate for all dimensions to 1.

[0134] At this point, a user request arrives at the system. The XDP probe captures the network packet event of this request at the network interface, extracts the trace_id, marks it as the network dimension, and generates a kernel timestamp. Based on the comprehensive load factor, the sampling rate of the network dimension is 1, and the event sampling passes. The event is stored in the 0th slot of the current CPU's circular event stack. The global causal index table is queried using trace_id as the key. The query result is empty, so a new causal context is created. The CPU number of the network dimension and slot index 0 are filled into the context, the initial alignment completion flag is set to 0, and the creation timestamp is recorded. The context is inserted into the global causal index table. Based on the event timestamp and the time wheel granularity of 100 microseconds, the slot number is calculated to be 5, and the context pointer is inserted at the head of the linked list of the 5th slot of the time wheel. The event count of this slot becomes 1, which does not exceed the event density threshold of 1000, so the time wheel is not triggered to advance immediately.

[0135] Subsequently, the request is processed by the target process. The syscall probe captures the read system call event, extracts the same trace_id, and marks it as a system dimension. The sampling rate is 1, and event sampling is successful. The event is stored in the first slot of the current CPU's circular event stack. The global causal index table is queried using the trace_id as the key to find the corresponding causal context. The context is updated, filling in the CPU number and slot index 1 for the system dimension. It is checked whether all dimension events have arrived. At this point, application dimension and database dimension events have not yet arrived, and the alignment completion flag is still 0. The slot number calculated based on the event timestamp is still 5, and the context is already in the linked list of the 5th slot, so it does not need to be inserted again.

[0136] Next, the Uprobe probe captures the log event of the application processing the request, extracts the trace_id, and marks it as an application dimension. The sampling rate is 1, and event sampling is successful. The event is stored in the second slot of the current CPU's circular event stack. The causal context is updated, filling in the CPU number and slot index 2 for the application dimension. At this point, there are still database dimension events that have not arrived, and the alignment completion flag is still 0.

[0137] Then, the Uprobe probe captures the database access event, extracts the trace_id, and marks it as a database dimension. The sampling rate is 1, and event sampling is successful. The event is stored in the third slot of the current CPU's circular event stack. The causal context is updated, filling in the CPU number and slot index 3 for the database dimension. At this point, events for all dimensions have arrived, and the alignment completion flag is set to 1.

[0138] The dynamic timeout of the time wheel has expired, and the time wheel advances to the 5th slot. Traverse the context list for this slot to find the causal context for the trace_id. Check that the alignment completion flag is 1, satisfying the aggregation condition. Retrieve the CPU ID and slot index from the context, and read the data for the four events from the corresponding CPU's circular event stack. Sort the four events by timestamp to generate composite data. This composite data does not contain urgent events, has a normal priority, and is written to the current CPU's normal-level cache queue. Mark the four slots in the event stack as free, remove the context from the global causal index table, and remove the context pointer from the time wheel list.

[0139] Five seconds later, the reporting timer for the normal-level buffer queue is triggered, and all composite data in the queue is reported to user space in batches. The user-space consumer process reads the composite data from the Ring Buffer, parses it, stores it, and displays it.

[0140] As the volume of business requests gradually increases, the system's CPU utilization rises to 60%, memory utilization rises to 50%, the target process's CPU utilization rises to 60%, thread pool saturation rises to 70%, the event buffer backlog rises to 20% of its total capacity, and the event loss rate is 0. The load calculation module calculates the system load factor as 0.6 multiplied by 0.7 plus 0.5 multiplied by 0.3, equaling 0.57. The target process load factor is 0.6 multiplied by 0.6 plus 0.7 multiplied by 0.4, equaling 0.64. The collector load factor is 0.2 multiplied by 0.6 plus 0 multiplied by 0.4, equaling 0.12. The overall load factor is 0.57 multiplied by 0.2 plus 0.64 multiplied by 0.6 plus 0.12 multiplied by 0.2, equaling 0.522. The overall load factor of 0.522 is still less than the first threshold of 0.6, the system remains in the low load range, and all parameters remain unchanged.

[0141] As the volume of business requests continues to increase, the system's CPU utilization rises to 70%, memory utilization rises to 60%, the target process's CPU utilization rises to 70%, thread pool saturation rises to 80%, the event buffer backlog rises to 30% of its total capacity, and the event loss rate is 1%. The load calculation module calculates the system load factor as 0.7 x 0.7 + 0.6 x 0.3, which equals 0.67. The target process load factor is 0.7 x 0.6 + 0.8 x 0.4, which equals 0.74. The collector load factor is 0.3 x 0.6 + 0.01 x 0.4, which equals 0.184. The overall load factor is 0.67 x 0.2 + 0.74 x 0.6 + 0.184 x 0.2, which equals 0.6148. The overall load factor of 0.6148 is greater than the first threshold of 0.6 and less than the second threshold of 0.8, indicating that the system has entered a medium load range. The parameter adjustment module linearly adjusts the time wheel granularity to 100 plus (0.6148-0.6) / (0.8-0.6) multiplied by 100, which equals 107.4 microseconds. The maximum wait window is linearly adjusted to 500 plus (0.6148-0.6) / (0.8-0.6) multiplied by 500, which equals 537 microseconds. The event density threshold is linearly adjusted to 1000 minus (0.6148-0.6) / (0.8-0.6) multiplied by 500, which equals 963. The sampling rate for non-critical dimensions is linearly adjusted to 1 minus (0.6148-0.6) / (0.8-0.6) multiplied by 0.5, which equals 0.963. The sampling rate for critical dimensions remains at 1.

[0142] At this point, when new events arrive, events in non-critical dimensions will be sampled at a sampling rate of 0.963, ensuring that most events can still be collected. The time wheel granularity has been slightly increased, the maximum waiting window has been slightly extended, and the overhead of event processing has been reduced, preventing further increases in system load.

[0143] When the volume of business requests reaches its peak, the system's CPU utilization rises to 90%, memory utilization rises to 80%, the target process's CPU utilization rises to 90%, thread pool saturation rises to 100%, the event buffer backlog rises to 80% of its total capacity, and the event loss rate is 5%. The load calculation module calculates the system load factor as 0.9 multiplied by 0.7 plus 0.8 multiplied by 0.3, which equals 0.87. The target process load factor is 0.9 multiplied by 0.6 plus 1.0 multiplied by 0.4, which equals 0.94. The collector load factor is 0.8 multiplied by 0.6 plus 0.05 multiplied by 0.4, which equals 0.5. The overall load factor is 0.87 multiplied by 0.2 plus 0.94 multiplied by 0.6 plus 0.5 multiplied by 0.2, which equals 0.838. The overall load factor of 0.838 is greater than the second threshold of 0.8, and the system enters the high-load range. The parameter adjustment module sets the time wheel granularity to 200 microseconds, the maximum wait window to 1000 microseconds, and the event density threshold to 500. The sampling rate for important dimensions is set to 0.8, and the sampling rate for unimportant dimensions is set to 0.2.

[0144] At this point, the sampling rate for non-critical dimensions is significantly reduced, with only 20 events being collected. The sampling rate for critical dimensions is also reduced to 80%, resulting in a substantial decrease in the number of events collected and a significant reduction in the overhead of the data collection system. The time wheel granularity is increased to 200 microseconds, and the frequency of time wheel advancement is halved, further reducing the overhead of event processing. This ensures that even under high system load, the data collection system will not consume excessive resources, prioritizing the operation of the business system.

[0145] During peak business hours, an application exception occurred. The Uprobe probe captured this exception event, extracted the trace_id, marked it as an application dimension, and set an emergency flag. Based on the comprehensive load factor, the application dimension is considered an important dimension, with a sampling rate of 0.8. A random number of 0.3 was generated, which is less than 0.8, so the event sampling passed. The event was stored in the 10th slot of the current CPU's circular event stack. The global causal index table was queried using the trace_id as the key to find the corresponding causal context. This context was originally inserted into the 20th slot of the time wheel. The system removed this context from the linked list of the 20th slot and inserted it at the head of the linked list of the current 15th slot, marking it as urgent and pending. The time wheel was immediately advanced, and the time wheel pointer moved to the 15th slot. The context linked list of this slot was traversed, and the urgent context was processed first. The maximum wait window check was skipped, and the event data from the network, system, and application dimensions that had arrived were aggregated to generate composite data. This composite data contained the urgent event, with an urgent priority, and was immediately reported to user space via the Ring Buffer. Upon receiving the composite data, the user-state monitoring system immediately issues an alarm to notify maintenance personnel to handle the fault.

[0146] As peak business hours passed and the volume of business requests gradually decreased, the system's CPU utilization dropped to 80%, memory utilization to 70%, target process CPU utilization to 80%, thread pool saturation to 90%, event buffer backlog to 50% of total capacity, and event loss rate to 3%. The load calculation module calculated a comprehensive load factor of 0.7204, returning to the medium load range. The parameter tuning module adjusted the time wheel granularity to 160 microseconds, the maximum wait window to 800 microseconds, the event density threshold to 600, and the sampling rate for non-critical dimensions to 0.7.

[0147] As the volume of business requests continued to decrease, the system's CPU utilization dropped to 60%, memory utilization to 50%, target process CPU utilization to 60%, thread pool saturation to 70%, event buffer backlog to 20% of total capacity, and event loss rate to 0%. The load calculation module calculated the overall load factor to be 0.522, returning to the low load range. The parameter adjustment module restored all parameters to their default values: time wheel granularity of 100 microseconds, maximum wait window of 500 microseconds, event density threshold of 1000, and sampling rate of 1 for all dimensions. The system returned to normal operation and continued to collect and process events.

[0148] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data sampling method based on multidimensional load adaptation, characterized in that, include: Acquire system load data, target process load data, and collector load data, and calculate the comprehensive load factor; In kernel mode, captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU, and the time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window. The time wheel granularity and maximum waiting window are dynamically adjusted based on the comprehensive load factor, and the composite data is reported to the user space for data sampling. Acquire system load data, target process load data, and collector load data; calculate the comprehensive load factor, including: The system layer's CPU utilization and memory utilization are collected as the system load data; the target process's CPU utilization and thread pool saturation are collected as the target process load data; and the current backlog and event loss rate of the event buffer are read as the collector load data. The system load data, target process load data, and collector load data are normalized respectively, and the comprehensive load factor is obtained by weighted summation according to preset weights. Dynamically adjusting the time wheel granularity and maximum wait window based on the comprehensive load factor includes: When the comprehensive load factor is less than the first threshold, the time wheel granularity is set to the first granularity, and the maximum waiting window is set to the first window. The first threshold is the upper limit of the system in a low load state. When the comprehensive load factor is greater than or equal to the first threshold and less than or equal to the second threshold, the time wheel granularity is linearly adjusted to between the first and second granularities, and the maximum waiting window is linearly adjusted to between the first and second windows. The second threshold is the lower limit of the system being in a high load state. When the comprehensive load factor is greater than the second threshold, the time wheel granularity is set to the second granularity, and the maximum waiting window is set to the second window, wherein the second granularity is greater than the first granularity, and the second window is greater than the first window.

2. The method according to claim 1, characterized in that, In kernel mode, captured multi-dimensional events are stored in a per-CPU circular event stack. Based on the time wheel timing index structure, events with the same tracking identifier are time-aligned to generate composite data, including: For any event in any dimension, extract the event's tracking identifier and dimension type, and store the event in the per-CPU circular event stack corresponding to the current CPU; Use the tracking identifier as the key to query the global causal index table, update or create the causal context, where the global causal index table is a hash table that stores the mapping relationship between tracking identifiers and corresponding event location metadata; The causal context is inserted into the corresponding slot of the time wheel time sequence index structure according to the event timestamp. When all the events of all dimensions corresponding to the causal context have arrived or exceeded the maximum waiting window, the corresponding events are aggregated to generate composite data.

3. The method according to claim 1, characterized in that, Reporting the composite data to user space includes: The composite data is divided into different levels according to its priority. Write composite data of different levels into the corresponding per-CPU cache queue; The composite data in each per-CPU cache queue is reported to user space in batches according to the time interval corresponding to the level.

4. The method according to claim 2, characterized in that, After inserting the causal context into the corresponding slot of the time wheel sequence index structure based on the event timestamp, the method further includes: Set a dynamic timeout based on the current slot's event count and historical event arrival rate; When the event count of any slot exceeds the event density threshold or there is an urgent pending context, the time wheel sequence index structure is immediately advanced, the causal context linked list of the current slot is traversed, and the alignment operation is performed.

5. The method according to claim 2, characterized in that, Also includes: When an event with an emergency flag is captured, query the corresponding causal context; The causal context is removed from the linked list of the slot in the original time wheel sequence index structure and inserted into the head of the linked list of the current slot, triggering the time wheel sequence index structure to advance immediately and prioritize the processing of the causal context.

6. The method according to claim 2, characterized in that, Aggregate corresponding events to generate composite data, including: Obtain the CPU number and slot index corresponding to each dimension of the event from the causal context; Read event data from the per-CPU circular event stack of the corresponding CPU based on the CPU number and slot index; Sort all event data by timestamp to generate composite data.

7. The method according to claim 1, characterized in that, Before the kernel stores the captured multi-dimensional events into the per-CPU circular event stack, the following steps are also included: The sampling rate for the corresponding dimension is determined based on the comprehensive load factor. The captured events are sampled based on the sampling rate; The events that pass sampling are stored in the per-CPU circular event stack.

8. A data sampling device based on multidimensional load adaptation, characterized in that, include: The load calculation module is used to acquire system load data, target process load data, and collector load data, and calculate the comprehensive load factor. Acquire system load data, target process load data, and collector load data; calculate the comprehensive load factor, including: The system layer's CPU utilization and memory utilization are collected as the system load data; the target process's CPU utilization and thread pool saturation are collected as the target process load data; and the current backlog and event loss rate of the event buffer are read as the collector load data. The system load data, target process load data, and collector load data are normalized respectively, and the comprehensive load factor is obtained by weighted summation according to preset weights. The kernel-mode alignment module is used to store captured multi-dimensional events into a per-CPU circular event stack. Based on the time wheel timing index structure, it performs timing alignment on events with the same tracking identifier to generate composite data. The per-CPU circular event stack refers to a fixed-size pre-allocated array structure corresponding to each CPU, and the time wheel timing index structure refers to an array structure that stores the causal context pointers to be aligned within the corresponding time window. The parameter adjustment module is used to dynamically adjust the time wheel granularity and the maximum waiting window based on the comprehensive load factor. Dynamically adjusting the time wheel granularity and maximum wait window based on the comprehensive load factor includes: When the comprehensive load factor is less than the first threshold, the time wheel granularity is set to the first granularity, and the maximum waiting window is set to the first window. The first threshold is the upper limit of the system in a low load state. When the comprehensive load factor is greater than or equal to the first threshold and less than or equal to the second threshold, the time wheel granularity is linearly adjusted to between the first and second granularities, and the maximum waiting window is linearly adjusted to between the first and second windows. The second threshold is the lower limit of the system being in a high load state. When the comprehensive load factor is greater than the second threshold, the time wheel granularity is set to the second granularity, and the maximum waiting window is set to the second window, wherein the second granularity is greater than the first granularity, and the second window is greater than the first window; The output module is used to report the composite data to the user space.

Images