System bottleneck root cause positioning method and system

Abstract

According to the system bottleneck root cause positioning method and system provided by the invention, through calculation of the criticality index, a key thread which causes abnormal performance of the system bottleneck is quickly identified. And constructing a dynamic waiting graph containing a waiting relationship between the threads, and finding out a sleep thread causing blockage in combination with the key thread and the waiting graph. And performing deep backtracking along a dependency chain of the sleep thread, determining a root cause thread which has the greatest influence on the overall performance based on the calculated root cause score, performing accurate CPU hotspot portraying on behaviors of the root cause thread in a resource holding period, and directly positioning a performance bottleneck to a specific source code. According to the scheme, hardware does not need to be customized, the application range is wide, a clear causal chain can be obtained, the threshold of a developer for understanding and repairing complexity performance problems is lowered, source codes can be directly positioned, and a complete attribution solution is achieved.

Description

Technical Field

[0001] This invention relates to the field of system performance optimization technology, and more specifically, to a method and system for locating the root causes of system bottlenecks. Background Technology

[0002] With the widespread adoption of multi-core CPUs and the development of parallel computing technology, modern software systems, especially high-performance applications such as databases, distributed services, and scientific computing, commonly employ multi-threaded models to fully utilize hardware resources and improve system throughput and response speed. However, inter-thread interaction and resource contention also introduce new performance complexities. Increased parallelism does not always lead to linear performance growth; once the system reaches a certain critical point, performance often saturates or even declines due to various bottlenecks. Therefore, accurately and efficiently locating and resolving these performance bottlenecks has become a core challenge in software engineering, particularly in performance engineering.

[0003] In the field of multithreaded performance optimization, existing bottleneck analysis techniques are mainly divided into three categories, but they all have the fundamental flaw of failing to simultaneously satisfy the deep integration of "computational" and "waiting" bottlenecks, automatic root cause discovery, quantitative attribution, and precise source code-level positioning.

[0004] Specifically, existing bottleneck analysis techniques mainly include user-mode instrumentation analysis, kernel event analysis, and hardware critical path detection. User-mode instrumentation analysis, such as Intel VTune and flame graphs, primarily involves embedding probes in user code to sample CPU execution hotspots. This method excels at on-CPU analysis, directly locating the code consuming the most CPU time. However, this approach is limited to a single thread and cannot explain "off-CPU" blocking bottlenecks between threads caused by lock contention, I / O waits, etc. Kernel event analysis, such as Linux perf and offcputime, analyzes the state of threads waiting for resources (off-CPU) by listening to kernel scheduling events (such as context switching and wake-ups). This method is transparent to the application, has low overhead, and accurately reflects system-level thread scheduling and waiting behavior. However, while it can identify "who is waiting" and even "who woke them up," it struggles to automatically identify the root cause of the blocking (the "bottleneck maker"), especially in complex scenarios where multiple threads interact. Additionally, hardware-based critical path detection methods, such as Criticism Stacks, identify critical threads in the system in real time using custom hardware performance counters (PMUs). This approach boasts an advanced theoretical model and extremely low overhead, making it suitable for real-time hardware scheduling optimization. However, it relies on custom hardware, lacks versatility and portability, and cannot be deployed on existing commercial servers. Furthermore, it suffers from coarse-grained analysis, typically providing only information on "which thread is critical," failing to offer source code-level information that developers can directly optimize.

[0005] It is evident that existing analytical techniques either only show "busyness" without seeing "waiting," or only show "waiting" without tracing the "root cause," or rely on custom hardware. Therefore, the industry urgently needs a new, non-intrusive, low-overhead solution that can deeply integrate "computational" and "waiting" bottlenecks and automatically complete the full attribution from "phenomenon" to "source code-level root cause." Summary of the Invention

[0006] The purpose of this invention is to provide a system for locating system bottlenecks and a system for achieving a complete attribution solution.

[0007] In a first aspect, the present invention provides a method for locating the root cause of a system bottleneck, the method comprising: Deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; Based on the detected thread switching events, the criticality of each thread is determined, and based on the criticality of each thread, the critical thread when a system bottleneck occurs is identified. Construct a waiting graph using each thread as a thread node and the waiting relationships as directed edges; The sleep thread is determined based on the key thread, the wait graph is queried based on the sleep thread to determine the thread node within the sleep thread time, and the root cause thread is determined based on the root cause score of the thread node. Query the call stack sample during the period when the root cause thread holds the resource, and locate the code information that causes the system bottleneck based on the call stack sample.

[0008] In an optional implementation, the step of determining the criticality of each thread based on the detected thread switching events includes: When a thread switching event is detected, record the current timestamp; The time interval is obtained based on the current timestamp and the timestamp of the last switching event; Query the number of currently active threads globally to obtain the degree of parallelism; The criticality increment is calculated based on the time interval and parallelism, and the criticality increment is accumulated into the accumulator of the thread to obtain the criticality.

[0009] In an optional implementation, the step of determining the critical thread when a system bottleneck occurs based on the criticality of each thread includes: Sort the criticality of each thread according to a set order; Based on the sorted threads, the thread with the highest criticality is identified as the critical thread when a system bottleneck occurs.

[0010] In an optional implementation, the step of constructing a waiting graph using each of the threads as thread nodes and waiting relationships as directed edges includes: When a wait start event is detected, it is marked as a pending match and a timeout timer is started. When the corresponding wait end event or wake-up event is received, it is matched by thread ID and resource ID, and the waiting time is calculated to generate a complete waiting interval. When a new waiting interval arrives, determine whether there is a corresponding thread node in the existing waiting graph. If not, create the corresponding thread node in the waiting graph and construct the edge information of the directed edge of the thread node based on the waiting relationship. If so, the edge information of the directed edge of the corresponding thread node is updated based on the waiting relationship; When the update of the waiting graph meets the preset conditions, the final waiting graph is generated.

[0011] In an optional implementation, the thread node includes a waiter node and a holder node, and a directed edge points from the waiter node to the holder node; The step of generating the final waiting graph when the update of the waiting graph meets the preset conditions includes: For each holder node, iterate through all waiter nodes that point to that holder node and calculate the contribution score of each waiter node to the holder node; For each waiter node, its total contribution score is distributed to all its waiting holder nodes according to the weight ratio; Calculate the final score received by each holder node, and when the final scores received by all holder nodes reach convergence or the number of iterations reaches a preset number, determine that the update of the waiting graph meets the preset conditions, and generate the final waiting graph.

[0012] In an optional implementation, the step of determining the sleeping thread based on the key thread includes: Threads that are in a sleeping state during the execution of the critical thread are identified as sleeping threads.

[0013] In an optional implementation, the step of determining the root cause thread based on the root cause score of the thread node includes: The final score is obtained based on all the contribution scores obtained by the thread node, and is used as the root cause score. The contribution scores are allocated to the waiter nodes pointing to the thread node according to the weight ratio. The thread node with the highest root cause score is identified as the root cause thread.

[0014] In an optional implementation, the step of locating the code information causing the system bottleneck based on the call stack sample includes: The probe is triggered on all CPUs at a fixed frequency to count the number of times the call stack samples are collected. The code in the call stack sample with the highest number of collections is identified as hot code, and all hot code is stored. After identifying the root cause thread, the corresponding call stack sample is found based on the thread number of the root cause thread and the timestamp, and the hot code causing the system bottleneck is identified.

[0015] Secondly, the present invention provides a system bottleneck root cause localization system, the system comprising: The deployment module is used to deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; The deployment module is used to deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; The determination module is used to determine the criticality of each thread based on the detected thread switching events, and to determine the critical thread when a system bottleneck occurs based on the criticality of each thread. The construction module is used to construct a waiting graph with each of the aforementioned threads as thread nodes and waiting relationships as directed edges; The query module is used to determine the sleeping thread based on the key thread, query the wait graph based on the sleeping thread to determine the thread node during the sleeping thread time, and determine the root cause thread based on the root cause score of the thread node. The localization module is used to query the call stack sample during the period when the root cause thread holds resources, and locate the code information that causes the system bottleneck based on the call stack sample.

[0016] This invention provides a method and system for locating system bottlenecks. By calculating criticality indicators, it quickly identifies critical threads causing abnormal performance and bottlenecks. A dynamic wait graph containing waiting relationships between threads is constructed. By combining the critical threads with the wait graph, the sleeping thread causing the blockage is identified. Deep backtracking is performed along the dependency chain of the sleeping thread, and based on the calculated root cause score, the root cause thread with the greatest impact on global performance is determined. The behavior of the root cause thread during resource holding is precisely profiled using CPU hotspot analysis, directly pinpointing the performance bottleneck to the specific source code. This solution requires no customized hardware, has a wide range of applications, provides a clear causal chain, lowers the barrier for developers to understand and fix complex performance problems, and directly locates the source code, achieving a complete attribution solution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a system bottleneck root cause localization method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of multiple types of probes deployed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the waiting event matching logic in an embodiment of the present invention; Figure 4 This is a schematic diagram of the waiting graph construction architecture in an embodiment of the present invention; Figure 5 This is a schematic diagram of the diagnostic report generation mechanism in an embodiment of the present invention; Figure 6 A functional block diagram of the system bottleneck root cause localization system provided in the embodiments of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0020] Please see Figure 1 This is a flowchart illustrating the system bottleneck root cause localization method provided in an embodiment of the present invention. The system bottleneck root cause localization method is executed on a computer device.

[0021] It should be understood that in other embodiments, the order of some steps in the system bottleneck root cause localization method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the system bottleneck root cause localization method are described below.

[0022] S11 deploys various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time.

[0023] S12, based on the detected thread switching events, determine the criticality of each thread, and based on the criticality of each thread, determine the critical thread when a system bottleneck occurs.

[0024] S13, construct a waiting graph with each thread as a thread node and the waiting relationship as a directed edge.

[0025] S14, determine the sleeping thread based on the key thread, query the waiting graph based on the sleeping thread to determine the thread node within the sleeping thread's time period, and determine the root cause thread based on the root cause score of the thread node.

[0026] S15, query the call stack sample during the period when the root cause thread holds the resource, and locate the code information that causes the system bottleneck based on the call stack sample.

[0027] The localization method provided in this embodiment is a system bottleneck root cause localization method based on thread parallelism and wait graph analysis. This method utilizes the Linux kernel's eBPF (Extended Berkeley Packet Filter) framework to collect system-level thread scheduling and synchronization events in real time and non-intrusively with extremely low performance overhead.

[0028] In this embodiment, multiple types of probes (eBPF probes) are deployed in the kernel's thread scheduling and synchronization events to achieve full coverage monitoring of various waiting events in the system.

[0029] The deployed probes include at least the following types: User-mode lock monitoring probe: Deploy a kprobe probe on the "futex_wait_queue_me" function to capture contention and wake-up events of POSIX lock primitives such as pthread_mutex_lock and pthread_cond_wait.

[0030] Kernel lock monitoring probes: Deploy probes on functions such as "_mutex_lock_slowpath" (mutex slow path), "rwsem_down_read_slowpath" (read / write semaphore read lock wait), "rwsem_down_write_slowpath" (read / write semaphore write lock wait), and "rt_mutex_slowlock" (real-time mutex wait). These probes are only triggered when actual lock contention occurs, ensuring accurate capture of blocking events.

[0031] Scheduling and wake-up monitoring probes: By attaching probes to the “sched_switch” and “sched_wakeup” events via tracepoints, the context switching and wake-up events of threads can be monitored to confirm changes in the sleep state of threads and wake-up information.

[0032] Other wait monitoring probes: Deploy probes on functions related to various types of wait events, such as I / O and network, to capture non-lock contention blocking events such as disk I / O waits, network transmission waits, and memory allocation waits.

[0033] All probes can accurately record nanosecond-level timestamps, thread IDs, resource identifiers (lock addresses, file handles, socket descriptors, etc.), and context information.

[0034] Various types of probes have corresponding specialized processors, such as Figure 2 As shown in the diagram, each processor stores the detected events into a unified event queue for subsequent event matching via the event matching engine.

[0035] Each event captured by an eBPF probe contains a unified data structure, which includes an event timestamp (nanosecond precision), event type (wait start, wait end, wake-up), thread ID, wait type (user-mode lock, kernel mutex, read-write lock, etc.), resource identifier (lock address, file handle, etc.), and additional context data. This unified data format ensures consistency and efficiency in subsequent processing.

[0036] In this embodiment, a criticality metric, PMetric (Parallelism Metric), is designed to identify critical threads that cause low system parallelism. Specifically, when a thread switching event is detected by a deployed probe, the criticality of that thread is calculated, and then the critical threads are determined based on the criticality of all threads.

[0037] In this embodiment, the step of determining the criticality of each thread based on the detected thread switching events can be implemented in the following way: When a thread switching event is detected, the current timestamp is recorded; the time interval is obtained based on the current timestamp and the timestamp of the last switching event; the number of threads currently in a globally active state is queried to obtain the parallelism; the criticality increment is calculated based on the time interval and the parallelism, and the criticality increment is accumulated into the accumulator of the thread to obtain the criticality.

[0038] In this embodiment, each time a sched_switch event occurs, the eBPF probe first obtains the current timestamp in nanoseconds using "bpf_ktime_get_ns()". Then, it subtracts the current timestamp from the globally stored timestamp of the previous switch event to obtain the time interval ΔT. Simultaneously, the probe queries a globally active thread counter maintained through atomic operations to obtain the number of threads currently in a globally active state, which is used as the parallelism N_active.

[0039] For each thread, the system maintains an independent PMetric accumulator. When a thread is scheduled off the CPU, the quotient ΔT divided by N_active is calculated as the criticality increment. This criticality increment is added to the thread's Pmetric accumulator to obtain the thread's current criticality. The physical meaning of this value is the average share of CPU time that thread "exclusively" occupies during its execution. If a thread's PMetric value is significantly higher than other threads, it indicates that the system's parallelism is very low during its execution, potentially indicating a serialization bottleneck.

[0040] Based on this, the steps for determining the critical thread when a system bottleneck occurs, based on the criticality of each thread, can be implemented in the following way: The criticality of each thread is sorted according to a set order; based on the sorted threads, the thread with the highest criticality is determined as the critical thread when a system bottleneck occurs.

[0041] In this embodiment, the criticality of each thread globally is sorted according to a set order, such as from largest to smallest or from smallest to largest. The thread with the highest criticality indicates that it has the largest average share of CPU time during execution, meaning that the system's parallelism is very low when this thread is executing. This thread is identified as a critical thread and will become the focus of subsequent in-depth analysis.

[0042] Simultaneously, a wait-for graph (W4G) is constructed based on the waiting relationships between threads. Each thread is treated as a node, and the waiting relationships are represented by directed edges. Specifically, the wait graph is constructed as follows: Upon detecting a wait start event, it is marked as a pending match state, and a timeout timer is started. When a corresponding wait end event or wake-up event is received, the thread ID and resource ID are matched, and the wait duration is calculated to generate a complete wait interval. When a new wait interval arrives, it is determined whether the existing wait graph has a corresponding thread node. If not, the corresponding thread node is created in the wait graph, and the edge information of the directed edges of the thread node is constructed based on the wait relationship. If it does exist, the edge information of the directed edges of the corresponding thread node is updated based on the wait relationship. When the update of the wait graph meets the preset conditions, the final wait graph is generated.

[0043] In this embodiment, the system maintains a complex event matching engine in user space, which is responsible for assembling discrete events collected from the kernel into complete waiting intervals.

[0044] Please refer to the following: Figure 3 When the kernel eBPF probe detects a WAIT_START event, it sends the WAIT_START event to the event queue. The event queue retrieves the events to be matched and sends them to the matching engine. The matching engine marks the events as pending matching and triggers the timeout handler to start a timeout timer. The matching engine checks if there is a matching WAIT_END event. If a matching event is found, it calculates the precise wait duration, generates the complete wait interval, and then updates the wait graph structure. If no match is found within the timeout period, the timeout handling is initiated, and expired unmatched events are cleaned up.

[0045] When the probe detects a wake-up event, the event queue triggers a matching wake-up relationship, and the matching engine establishes a waiter-holder connection relationship corresponding to the wake-up relationship between threads.

[0046] See also Figure 4In this embodiment, the system supports thirteen different types of wait events, including user-mode locks, kernel mutexes, read-write locks, I / O waits, network waits, and disk waits. Each type has its specific event matching rules and timeout handling strategies. For example, for futex locks, the matching rule is the same thread ID and the same lock address; for I / O waits, the matching rule is the same thread ID and the same file descriptor. Figure 4 In this process, different types of events are processed uniformly through an event classifier and then managed separately by the ThreadNode manager, WaitEdge manager, and virtual resource node manager.

[0047] During the construction and dynamic maintenance of nodes and edges in the waiting graph, the waiting graph adopts an adjacency list data structure. Each thread corresponds to a thread node, and each waiting relationship corresponds to a directed edge. When a new waiting interval arrives, the system checks whether there is a corresponding thread node in the existing waiting graph. If not, a new thread node is created. Furthermore, it checks whether there are duplicate directed edges. If so, its statistics (cumulative waiting time, number of waits, etc.) are updated; otherwise, a new directed edge is created. The system also implements an efficient expired edge cleanup mechanism, periodically deleting directed edges that have not been updated for a long time to prevent memory leaks.

[0048] In this embodiment, the thread nodes in the waiting graph include waiter nodes and holder nodes, and directed edges point from waiter nodes to holder nodes. During the iterative update process of the waiting graph, when the update of the waiting graph meets preset conditions, the final waiting graph is generated. Specifically, this step is implemented in the following way: For each holder node, traverse all waiter nodes pointing to that holder node and calculate the contribution score of each waiter node to the holder node; for each waiter node, distribute its total contribution score to all its waiting holder nodes according to the weight ratio; calculate the final score received by each holder node, and when the final scores received by all holder nodes reach convergence or the number of iterations reaches a preset number, determine that the update of the waiting graph meets the preset conditions and generate the final waiting graph.

[0049] Unlike traditional PageRank, this embodiment uses the Root Cause Score (RCS) algorithm to construct the inlink structure. For each directed edge W→H (thread node W points to thread node H), W is added to the inlinks[H] list of H. The advantage of this reverse construction is that each holder node directly knows all the waiter nodes waiting for it, without needing to recalculate the transpose of the graph in each iteration.

[0050] Furthermore, a time-based weighting method is adopted, where the weight of directed edges is directly based on the waiting time, with the formula w = wait_time. Here, wait_time comes from the nanosecond-level waiting time precisely measured by the eBPF probe. This simplified weighting strategy has the following advantages: (1) Intuition: The weight directly reflects the severity of the waiting, and the longer the waiting time of the directed edge, the greater the impact on the root cause analysis; (2) Purity: It avoids interference from other factors such as criticality PMetric, allowing the algorithm to focus on the waiting relationship itself; (3) Computational efficiency: It reduces the complexity of weight calculation and improves the execution efficiency of the algorithm.

[0051] The waiting graph updates and iterations use a three-level nested loop structure: outer loop, middle loop, and inner computation.

[0052] In the outer loop, all holder nodes H are traversed to ensure that every possible root cause node is considered. In the middle loop, for the current holder node H, all waiter nodes pointing to that holder node are traversed, that is, all waiter nodes in its inlinks[H] list are traversed.

[0053] In the inner calculation, the contribution score of each waiter node to the holder node is calculated, including weight normalization and score allocation. A score propagation mechanism is used, based on the principle of responsibility sharing. Each waiter node W allocates its current score old_score[W], i.e., its total contribution score, to all the holder nodes it is waiting for according to their weight proportions. The allocation formula is as follows: contribution = (w[W→H] / total_out_weight[W]) × old_score[W] In the above formula, the numerator represents the weight of waiter node W to holder node H, and the denominator represents the sum of the weights of W to all holder nodes.

[0054] In this embodiment, the score received by each holder node needs to be multiplied by a damping coefficient d (usually set to 0.85), so the final score is: new_score[H] = (1-d) / N + d × Σ(contribution). This ensures that the score can be propagated while maintaining the stability of the system.

[0055] In this embodiment, the L1 norm can be used to determine whether convergence has occurred. If convergence has occurred, it can be determined that the preset conditions are met. Alternatively, the preset number of times can be used to determine whether the preset conditions are met.

[0056] For each holder node, the difference between the final scores of the two iterations is calculated to obtain the change. The total change is obtained by summing the changes of all holder nodes, which is error = Σ|new_scores[i] - old_scores[i]|, that is, the sum of the absolute values ​​of the difference between the new and old score vectors.

[0057] The L1 norm reflects the total change in the fraction vector. When the total change is less than a preset threshold, such as error < ε (e.g., 1e-6), it indicates that the changes in the fractions of all nodes are very small, and the algorithm is close to a stable state.

[0058] In this embodiment, the L1 norm is not sensitive to outliers and is more suitable than the L2 norm for handling fractional distributions in sparse graph structures.

[0059] The L1 norm belongs to the power iteration method, and the mathematical basis of the power iteration method is as follows: Iteration formula: x^{(k+1)} = (1-d) / N × 1 + d × M × x^{(k)}, where M is the transition matrix and x is the fraction vector.

[0060] Since the RCS matrix is ​​a random matrix and satisfies the conditions of irreducibility and aperiodicity, according to the Perron-Frobenius theorem, the power iteration method must converge to a unique principal eigenvector.

[0061] The convergence speed depends on the second largest eigenvalue. With principal eigenvalues The ratio. The damping coefficient d=0.85 ensures... <0.85, typically achieving machine accuracy convergence within 20-30 iterations.

[0062] In this embodiment, a dual protection mechanism is set, namely, a precision threshold (L1 norm < ε) and a maximum iteration limit (usually 50 rounds). The iteration ends when one of the conditions is met, which can ensure that the algorithm completes within a limited time.

[0063] Based on the identification of the critical thread, the sleep thread is determined accordingly. This step can be achieved in the following way: Identify threads that are in a sleeping state during the execution of critical threads and designate them as sleeping threads.

[0064] Once the system identifies the critical thread and its active time period, it needs to find all threads that are blocked during that active time period. The system uses an efficient interval overlap algorithm to compare the active time period of the critical thread with all waiting intervals recorded in the waiting graph. If a waiting interval overlaps with the active time period of the critical thread, the corresponding waiting thread is marked as an affected thread, i.e., a sleeping thread.

[0065] Based on the identified sleeping threads, the waiting graph is queried to determine the thread nodes within the sleeping thread's time frame.

[0066] In this embodiment, for each affected sleeping thread, the system precisely determines the cause of its blocking by using the edge information of the directed edges in the wait graph, including the type of resource being waited for (lock, I / O, network, etc.), the resource identifier, and the thread holding the resource. In this way, the system can construct a complete blocking dependency chain, tracing back from the victim to the root cause.

[0067] That is, the thread node within the sleep thread's time frame is determined based on the blocking dependency chain, and the root cause thread is determined based on the root cause score of that thread node. Specifically, this step can be implemented in the following way: The final score is obtained based on all the contribution scores obtained by the thread node, which is used as the root cause score. The contribution scores are allocated to the waiter nodes pointing to the thread node according to the weight ratio. The thread node with the highest root cause score is determined as the root cause thread.

[0068] In this embodiment, for each affected sleeping thread, there may be more than one thread node during its sleep time, that is, there may be more than one blocking dependency chain. In this case, it is necessary to determine which of the more than one thread node in the blocking dependency chain is the root cause node.

[0069] In this embodiment, for thread nodes within the sleep thread's time period, their root cause scores can be calculated using the same method as the final score calculation for the holder nodes described above. The thread node with the highest root cause score is selected as the root cause thread affecting the sleep thread.

[0070] After identifying the root cause thread, query the call stack samples during the period when the root cause thread held the resources. Based on the call stack samples, locate the code information causing the system bottleneck. Specifically, this step can be implemented in the following way: The probe is triggered on all CPUs at a fixed frequency to count the number of times the call stack samples are collected. The code in the call stack sample with the highest number of collections is identified as hot code, and all hot code is stored. After the root cause thread is identified, the corresponding call stack sample is found based on the thread number of the root cause thread and the timestamp, and the hot code causing the system bottleneck is identified.

[0071] Combination Figure 5As shown in this embodiment, the probe is triggered on all CPUs at a fixed frequency. This allows for the counting of times call stack samples are collected, enabling a frequency-based clustering algorithm. The code in the call stack sample collected most frequently is identified as hotspot code. All hotspot code data is stored. Finally, when determining the root cause thread, the thread ID of the root cause thread is combined with the time to find the corresponding call stack sample and hotspot code. In this way, the hotspot code causing the system bottleneck can be identified.

[0072] Furthermore, based on the identified hot code, a statistical report of hot functions can be generated, along with relevant optimization suggestions, and finally, a structured diagnostic report can be output.

[0073] In this embodiment, the generated structured diagnostic report can contain multiple levels of diagnostic information: at the macro level, it displays the distribution and trend of criticality (PMetric) of key threads; at the micro level, it displays specific wait dependency chains and root cause threads; and at the code level, it displays the hot execution paths of root cause threads and possible optimization suggestions. The report can be stored in JSON format for easy subsequent automated processing and visualization.

[0074] The system implements a real-time visualization interface based on web technology, which can dynamically display the topology of the waiting graph, the distribution of root cause scores (RCS), and the trend over time. The graph uses a force-directed layout algorithm to automatically adjust the node positions, highlighting root cause nodes with high RCS scores and major blocking paths.

[0075] In summary, this solution begins by using metrics such as PMetrics to quickly identify critical threads exhibiting abnormal behavior due to low parallelism and their problematic time windows. The system then correlates this window with the global wait graph (W4G) to identify all concurrent sleeping threads during this period and performs deep backtracking along the dependency chain of "waiter → resource → occupier." Through an innovative root cause propagation algorithm (RCS), the system automatically calculates and sorts the root cause threads with the greatest impact on global performance, rather than simply identifying "wake-ups." Finally, a precise CPU hotspot profile is created based on the behavior of the root cause thread during resource occupancy. By using call stack sampling and symbol table resolution, the performance bottleneck is directly located to specific source code line numbers, ultimately generating a complete diagnostic report from macroscopic phenomena to microscopic code details.

[0076] The system bottleneck root cause localization method provided in this embodiment can achieve macro-level parallelism assessment. Through a novel criticality index PMetric, it can quantify in real time the degree of "exclusivity" of each thread on the system's parallel resources during its execution, and quickly identify the "critical threads" that cause low system efficiency due to uneven load or long serial execution.

[0077] In addition, it enables micro-level waiting relationship tracing, accurately depicting the "wait-wake" dependency relationship between threads by constructing a dynamic waiting graph with rich context.

[0078] Furthermore, by using an original Root Cause Propagation (RCS) algorithm, the information from the two dimensions mentioned above is fused to calculate the "systematic inhibition score" of each potential bottleneck, thereby automatically and accurately locating the root cause thread that has the greatest impact on the overall system performance and is most worthy of optimization.

[0079] Compared with existing technologies, this solution has the advantages of being non-invasive and low-overhead, having high positioning accuracy, enabling root cause self-discovery, being highly interpretable, and having a wide range of applications.

[0080] Specifically, the system is entirely based on the Linux eBPF framework, requiring no modification to the kernel or user programs, with average performance overhead controlled within 5%, and can run directly and continuously in a production environment.

[0081] The system employs proprietary PMetric and RCS algorithms, enabling it to automatically distinguish between "phenomena" and "root causes," accurately identify the "first bottleneck" that has the greatest impact on system performance, and directly pinpoint the specific line of code that causes the bottleneck.

[0082] The final diagnostic report presents a clear cause-and-effect chain, not only telling developers "where it's slow," but also explaining "why it's slow" and "who affected whom," greatly lowering the barrier for developers to understand and fix complex performance issues.

[0083] Because it operates at the operating system kernel level, this method is independent of upper-level programming languages ​​and naturally supports almost all applications running on Linux, such as C / C++, Go, Rust, Java, and Python. It is especially suitable for high-concurrency and high-complexity scenarios such as databases, distributed systems, and microservice architectures.

[0084] Based on the same inventive concept, please refer to Figure 6 This invention also provides a functional module diagram of a system bottleneck root cause localization system. This embodiment can divide the system bottleneck root cause localization system into functional modules based on the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this invention embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0085] For example, when dividing functional modules according to their respective functions, Figure 6 The system bottleneck root cause localization system shown is only a schematic diagram of a device. The system bottleneck root cause localization system may include a deployment module, a determination module, a construction module, a query module, a second determination module, and a localization module. The functions of each module of this system bottleneck root cause localization system are described in detail below.

[0086] The deployment module is used to deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; The determination module is used to determine the criticality of each thread based on the detected thread switching events, and to determine the critical thread when a system bottleneck occurs based on the criticality of each thread. The construction module is used to construct a waiting graph with each of the aforementioned threads as thread nodes and waiting relationships as directed edges; The query module is used to determine the sleeping thread based on the key thread, query the wait graph based on the sleeping thread to determine the thread node during the sleeping thread time, and determine the root cause thread based on the root cause score of the thread node. The localization module is used to query the call stack sample during the period when the root cause thread holds resources, and locate the code information that causes the system bottleneck based on the call stack sample.

[0087] The system bottleneck root cause localization system provided in this embodiment can be used to execute the system bottleneck root cause localization method under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0088] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0089] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0091] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0093] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for locating the root cause of a system bottleneck, characterized in that, The method includes: Deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; Based on the detected thread switching events, the criticality of each thread is determined, and based on the criticality of each thread, the critical thread when a system bottleneck occurs is identified. Construct a waiting graph using each thread as a thread node and the waiting relationships as directed edges; The sleep thread is determined based on the key thread, the wait graph is queried based on the sleep thread to determine the thread node within the sleep thread time, and the root cause thread is determined based on the root cause score of the thread node. Query the call stack sample during the period when the root cause thread holds the resource, and locate the code information that causes the system bottleneck based on the call stack sample.

2. The system bottleneck root cause localization method according to claim 1, characterized in that, The step of determining the criticality of each thread based on the detected thread switching events includes: When a thread switching event is detected, record the current timestamp; The time interval is obtained based on the current timestamp and the timestamp of the last switching event; Query the number of currently active threads globally to obtain the degree of parallelism; The criticality increment is calculated based on the time interval and parallelism, and the criticality increment is accumulated into the accumulator of the thread to obtain the criticality.

3. The system bottleneck root cause localization method according to claim 1, characterized in that, The step of determining the critical thread when a system bottleneck occurs based on the criticality of each thread includes: Sort the criticality of each thread according to a set order; Based on the sorted threads, the thread with the highest criticality is identified as the critical thread when a system bottleneck occurs.

4. The system bottleneck root cause localization method according to claim 1, characterized in that, The step of constructing a waiting graph using each thread as a thread node and waiting relationships as directed edges includes: When a wait start event is detected, it is marked as a pending match and a timeout timer is started. When the corresponding wait end event or wake-up event is received, it is matched by thread ID and resource ID, and the waiting time is calculated to generate a complete waiting interval. When a new waiting interval arrives, determine whether there is a corresponding thread node in the existing waiting graph. If not, create the corresponding thread node in the waiting graph and construct the edge information of the directed edge of the thread node based on the waiting relationship. If so, the edge information of the directed edge of the corresponding thread node is updated based on the waiting relationship; When the update of the waiting graph meets the preset conditions, the final waiting graph is generated.

5. The system bottleneck root cause localization method according to claim 4, characterized in that, The thread node includes a waiter node and a holder node, and the directed edge points from the waiter node to the holder node; The step of generating the final waiting graph when the update of the waiting graph meets the preset conditions includes: For each holder node, iterate through all waiter nodes that point to that holder node and calculate the contribution score of each waiter node to the holder node; For each waiter node, its total contribution score is distributed to all its waiting holder nodes according to the weight ratio; Calculate the final score received by each holder node, and when the final scores received by all holder nodes reach convergence or the number of iterations reaches a preset number, determine that the update of the waiting graph meets the preset conditions, and generate the final waiting graph.

6. The system bottleneck root cause localization method according to claim 4, characterized in that, The step of determining the sleeping thread based on the key thread includes: Threads that are in a sleeping state during the execution of the critical thread are identified as sleeping threads.

7. The system bottleneck root cause localization method according to claim 1, characterized in that, The thread node includes a waiter node and a holder node, and the directed edge points from the waiter node to the holder node; The step of determining the root cause thread based on the root cause score of the thread node includes: The final score is obtained based on all the contribution scores obtained by the thread node, and is used as the root cause score. The contribution scores are allocated to the waiter nodes pointing to the thread node according to the weight ratio. The thread node with the highest root cause score is identified as the root cause thread.

8. The system bottleneck root cause localization method according to claim 1, characterized in that, The step of locating the code information causing the system bottleneck based on the call stack sample includes: The probe is triggered on all CPUs at a fixed frequency to count the number of times the call stack samples are collected. The code in the call stack sample with the highest number of collections is identified as hot code, and all hot code is stored. After identifying the root cause thread, the corresponding call stack sample is found based on the thread number of the root cause thread and the timestamp, and the hot code causing the system bottleneck is identified.

9. A system bottleneck root cause localization system, characterized in that, The system includes: The deployment module is used to deploy various types of probes in the kernel's thread scheduling and synchronization events to capture various types of thread events in real time; The determination module is used to determine the criticality of each thread based on the detected thread switching events, and to determine the critical thread when a system bottleneck occurs based on the criticality of each thread. The construction module is used to construct a waiting graph with each of the aforementioned threads as thread nodes and waiting relationships as directed edges; The query module is used to determine the sleeping thread based on the key thread, query the wait graph based on the sleeping thread to determine the thread node during the sleeping thread time, and determine the root cause thread based on the root cause score of the thread node. The localization module is used to query the call stack sample during the period when the root cause thread holds resources, and locate the code information that causes the system bottleneck based on the call stack sample.

Images