Application delay root cause analysis method, device, storage medium and electronic equipment

Abstract

The application relates to an application delay root cause analysis method, device, storage medium and electronic equipment, the method comprising the following steps: obtaining a tracking identifier of an application request, writing an application thread identifier and the tracking identifier into a first file; reading an entry pressure stall information snapshot of a target control group record according to a reading period, writing the snapshot and a control group identifier into a second file; obtaining the tracking identifier, the entry pressure stall information snapshot and entry time according to an entry probe, and writing into a third file; obtaining the tracking identifier, an exit pressure stall information snapshot and exit time based on an exit probe, querying the third file based on the tracking identifier, obtaining the entry pressure stall information snapshot and the entry time; and obtaining a pressure stall information change value and application delay based on the entry pressure stall information snapshot, the entry time, the exit pressure stall information snapshot, the entry time and the exit time, and determining a root cause in combination with a root cause threshold. The accuracy of root cause analysis can be ensured.

Description

Technical Field

[0001] This invention relates to the field of performance diagnostics technology, and more particularly to an application of a delay root cause analysis method, apparatus, storage medium, and electronic device. Background Technology

[0002] During the operation of an application (or app) on a cloud-native system, the application may run slowly or malfunction due to hardware resource scheduling of the cloud-native system or some logical defects in the design or development of the application, resulting in significant latency. Therefore, it is necessary to analyze the root causes that may cause significant latency in order to make corresponding improvements based on the root causes found in the analysis, so as to maintain the efficient operation of the cloud-native system.

[0003] In related technologies, cloud-native systems classify different applications into critical business services and configure dedicated control groups (cgroups) for each category of critical business services. These control groups monitor the operation of critical business services (applications) belonging to their respective groups and use the cumulative stall time of each application to characterize latency for resource scheduling, thereby ensuring the operation of the cloud-native system. However, this method, which uses cumulative stall time for resource scheduling, assumes that the root cause of stalling in critical business services within a control group is inadequate hardware resource scheduling. Therefore, when an application stalls, if hardware resources permit, more hardware resources are allocated to the application to reduce the stall. This approach does not consider stalling caused by logical flaws in the development and design of critical business services. Consequently, if a stall is caused by logical flaws in a critical business service, hardware resource scheduling will not resolve the underlying stall problem, resulting in low accuracy of root cause analysis and impacting the efficiency of resource scheduling. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, storage medium, and electronic device for analyzing the root causes of delays in applications.

[0005] Specifically, the present invention is achieved through the following technical solution: According to a first aspect of the present invention, an applied method for delayed root cause analysis is provided, the applied method for delayed root cause analysis comprising: Obtain the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; According to a pre-set reading cycle, read the ingress pressure stagnation information snapshot recorded by the target control group configured for the application request corresponding to the application request, and write the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a pre-set second data packet filter mapping file; Based on the ingress probe in the pre-set request probe, obtain the tracking identifier, ingress pressure stagnation information snapshot, and ingress time, and write them into the pre-set third packet filter mapping file; Based on the exit probe in the request probe, obtain the tracking identifier, exit pressure stagnation information snapshot and exit time; based on the tracking identifier, query the third data packet filter mapping file to obtain the entry pressure stagnation information snapshot and entry time corresponding to the tracking identifier. Based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time, the pressure stagnation information change value and application delay are obtained. Based on the pressure stagnation information change value, application delay, and a pre-set root cause threshold, the root cause is determined.

[0006] Optionally, obtaining the tracking identifier configured in the application request initiated by the user, and writing the application thread identifier executing the application request and the tracking identifier into a pre-set first packet filter mapping file, includes: The user-space tracking identifier injector is used to extract the configured tracking identifier at the entry point of the application request initiated by the user. Obtain the application thread identifier that executes the application request, and write it into the first data packet filter mapping file using the application thread identifier and the tracking identifier as the corresponding key values.

[0007] Optionally, obtaining the tracking identifier, inlet pressure stagnation information snapshot, and inlet time based on the inlet probe in the request probe includes: Call the bpf_get_current_pid_tgid function to obtain the identifier of the running thread; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the obtained current control group identifier, query the second data packet filter mapping file to obtain a snapshot of the inlet pressure stagnation information corresponding to the current control group identifier; Record the current entry time; Write the obtained snapshot of the inlet pressure stagnation information, the current inlet time, the tracking identifier, and the current control group identifier into the third data packet filter mapping file.

[0008] Optionally, the step of querying the third data packet filter mapping file based on the tracking identifier to obtain the ingress pressure stagnation information snapshot and ingress time corresponding to the tracking identifier includes: Call the bpf_get_current_pid_tgid function to obtain the process ID to which the running thread ID belongs; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the acquired tracking identifier and current control group identifier, query the third data packet filter mapping file to obtain the inlet pressure stagnation information snapshot and inlet time.

[0009] Optionally, the step of obtaining the pressure stagnation information change value and application delay based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time includes: Calculate the difference between the outlet pressure stagnation information snapshot and the inlet pressure stagnation information snapshot to obtain the pressure stagnation information change value; The application delay is obtained by calculating the time difference between the exit time and the entry time.

[0010] Optionally, the stress stagnation information change values ​​include: CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values. The step of determining the root cause based on the stress stagnation information change values, application latency, and a pre-set root cause threshold includes: Select the largest value from the CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values; Calculate the ratio of the maximum change in stagnant information to application latency; If the ratio is greater than the preset root cause threshold, the root cause label is determined to be the hardware resource contention corresponding to the maximum change value of the stagnant information. If the ratio is not greater than the preset root cause threshold, the root cause label is determined to be the application's logic.

[0011] Optionally, the method further includes: If the "some" field in the snapshot of export pressure stall information indicates that no task is stalled due to resource unavailability, then the read cycle is increased. If the full field in the snapshot of export pressure stagnation information indicates that all tasks are stagnant at the same time, then the read cycle is reduced.

[0012] The latency root cause analysis method applied in this technical solution obtains the tracking identifier configured in the application request initiated by the user, and writes the application thread identifier executing the application request and the tracking identifier into a pre-set first data packet filter mapping file; according to a pre-set reading cycle, it reads the ingress pressure stagnation information snapshot recorded by the target control group configured for the application corresponding to the application request, and writes the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a pre-set second data packet filter mapping file; based on the ingress probe in the pre-set request probe, it obtains the tracking identifier, ingress pressure stagnation information snapshot, and ingress time, and writes them into a pre-set third data packet filter mapping file; based on the egress probe in the request probe, it obtains the tracking identifier, egress pressure stagnation information snapshot, and egress time, and queries the third data packet filter mapping file based on the tracking identifier to obtain the ingress pressure stagnation information snapshot and egress time corresponding to the tracking identifier; based on the ingress pressure stagnation information snapshot, egress time, egress pressure stagnation information snapshot, egress time, and egress time, it obtains the pressure stagnation information change value and application latency, and determines the root cause based on the pressure stagnation information change value, application latency, and a pre-set root cause threshold. In this way, by setting a tracking identifier for the application and obtaining the ingress PSI snapshot, ingress time, outgress PSI snapshot, and outgress time based on the request probe, the pressure stagnation information change value and application latency corresponding to the tracking identifier are calculated. Based on the pressure stagnation information change value, application latency, and root cause threshold, the root cause of the application latency is determined. The root cause type of application request latency can be quantified according to the root cause threshold, which can effectively improve the accuracy of root cause analysis.

[0013] According to a second aspect of the present invention, an application-based delayed root cause analysis apparatus is provided, the application-based delayed root cause analysis apparatus comprising: The first mapping file acquisition module is used to acquire the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; The second mapping file acquisition module is used to read the ingress pressure stagnation information snapshot recorded by the target control group of the application configuration corresponding to the application request according to a preset reading cycle, and write the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a preset second data packet filter mapping file. The third mapping file acquisition module is used to acquire the tracking identifier, entry pressure stagnation information snapshot, and entry time based on the entry probe in the pre-set request probe, and write them into the pre-set third data packet filter mapping file; The parameter acquisition module is used to acquire the tracking identifier, the snapshot of the outlet pressure stagnation information, and the outlet time based on the outlet probe in the request probe, and to query the third data packet filter mapping file based on the tracking identifier to acquire the snapshot of the inlet pressure stagnation information and the inlet time corresponding to the tracking identifier. The delayed root cause determination module is used to obtain the pressure stagnation information change value and application delay based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time and outlet time, and determine the root cause based on the pressure stagnation information change value, application delay and a preset root cause threshold.

[0014] According to a third aspect of the invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a delay root cause analysis method for any possible implementation of the first aspect.

[0015] According to a fourth aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a delay root cause analysis method for any possible implementation of the first aspect. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a delayed root cause analysis method for an application provided in an embodiment of the present invention; Figure 2 Another flowchart illustrating a delayed root cause analysis method for an application provided by an embodiment of the present invention; Figure 3 This is another flowchart illustrating a delayed root cause analysis method for an application provided in an embodiment of the present invention. Figure 4 A schematic diagram of a delayed root cause analysis device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In related technologies, applications are categorized based on critical business services, and control groups are configured for each category of critical business services. Since cgroups typically do not contain other user-space business processes besides the target application (critical business service) main process and its kernel threads, the resource usage status within the cgroup can highly reflect the behavior of each application, thus creating a low-noise observation environment. Furthermore, since version 4.20, the Linux kernel has introduced Pressure Stall Information (PSI). The Information mechanism provides stall time statistics for three types of resources (CPU, MEM, and IO) through cgroup interfaces (such as / sys / fs / cgroup / ... / memory.pressure). The 'some' field indicates that at least one task (application) stalls due to resource unavailability; the 'full' field indicates that all tasks stall simultaneously; and the 'total' field indicates the cumulative stall time (in nanoseconds). Therefore, the 'total' field can effectively monitor application operation, enabling latency monitoring of each application based on the control group. Hardware resources can be scheduled to ensure the reliable operation of applications within the control group in the cloud-native system. However, while this method can accurately obtain latency (stall) caused by hardware resource contention, it does not consider stalls caused by logical flaws in the development and design of critical business services (applications). Besides lower accuracy in root cause analysis, it also has the following drawbacks: Granularity mismatch: PSI is a cgroup-level aggregated metric that applies to all applications within a cgroup and cannot be correlated to specific applications. It also only tracks the application layer and lacks linkage with kernel resource pressure. Root cause ambiguity: When application latency increases, it is difficult to distinguish whether the bottleneck is due to the application's logical bottleneck, such as a stagnation caused by a slow-designed structured query language (SQL), or stagnation caused by underlying resource contention, such as memory reclamation. Unable to quantify contribution: When application latency increases, even if resource pressure can be identified, it is not possible to know the specific duration of application downtime.

[0021] This embodiment provides a high-precision, low-overhead, and non-intrusive (or minimally intrusive) method for request-level resource contention root cause analysis. It combines the Linux kernel PSI mechanism with the extended Berkeley Packet Filter (eBPF). User-space actively writes trace identifiers (trace_id) into the BPF Map. Based on eBPF, the application captures and binds an entry PSI snapshot representing latency and the application's start time at the request entry point, and captures and binds an exit PSI snapshot and the application's end time at the request exit point, associating these with the trace_id to complete the ΔPSI calculation. The final output is a structured root cause report traceable to open telemetry. Furthermore, by setting up a single-process dedicated cgroup scenario, the PSI stall time can be precisely allocated to the specific requesting application, thereby obtaining root cause analysis for that application.

[0022] See Figure 1 This invention provides a method for analyzing the delayed root causes of an application, which may include the following steps: S101. Obtain the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; In this embodiment, as an optional embodiment, the user initiates an application request, such as an HTTP request, to run the application.

[0023] In this embodiment, as an optional embodiment, obtaining the tracking identifier configured in the application request initiated by the user, and writing the application thread identifier executing the application request and the tracking identifier into a pre-set first data packet filter mapping file includes: The user-space tracking identifier injector is used to extract the configured tracking identifier at the entry point of the application request initiated by the user. Obtain the application thread identifier that executes the application request, and write it into the first data packet filter mapping file using the application thread identifier and the tracking identifier as the corresponding key values.

[0024] In this embodiment, the user-space trace identifier injector, specifically the user-space trace_id injector, extracts the trace_id at the entry point of the user-initiated application request and writes it to the first BPF Map using the application thread identifier (tid, Thread ID) and trace_id as keys. The BPF Map is a key / value data structure stored in the kernel and can be accessed by BPF applications. By creating the BPF Map in the kernel application and returning the corresponding file descriptor, applications running in user space can access and manipulate the BPF Map through the returned file descriptor.

[0025] In this embodiment, a user-space trace_id injection is performed using a user-space trace identifier injector. This is achieved by parsing the application request at the application request entry point, for example, by obtaining the trace_id from the HTTP request header or context. As an optional embodiment, the first packet filter mapping file is a BPF Map "trace_map".

[0026] In this embodiment, as an optional implementation, the code segment for performing user-mode trace_id injection is as follows: / / At the beginning of the HTTP handler func handler(w http.ResponseWriter, r *http.Request) { traceID := r.Header.Get("traceparent") / / OpenTelemetry standard if traceID == "" { return} tid := unix.Gettid() / / Get OS thread ID bpfTraceMap.Update(uint32(tid), parseTraceID(traceID)) / / Write BPFMap "trace_map" / / ...business logic... } S102. According to a pre-set reading cycle, read the inlet pressure stagnation information snapshot recorded by the target control group configured for the application request, and write the inlet pressure stagnation information snapshot and the control group identifier of the target control group into a pre-set second data packet filter mapping file. In this embodiment, control groups are pre-configured for applications. As an optional embodiment, a dedicated control group is configured for each application. As another optional embodiment, a single control group is configured for multiple applications. This embodiment does not limit this approach. After the application starts, the corresponding control group monitors the application's operation and provides stall time statistics for three types of resources (CPU, MEM, and IO) through the cgroup interface using Pressure Stall Information Snapshot (PSI snapshot).

[0027] In this embodiment, based on a pre-set reading cycle, for example, 5ms, the cumulative stall time of the PSI snapshot in the target cgroup is read periodically, and the cumulative stall time is written into the second BPF Map using the control group identifier (cgroup ID) of the target control group as the key.

[0028] In this embodiment, as an optional embodiment, the cumulative stall time includes, but is not limited to: CPU cumulative stall time, memory (MEM) cumulative stall time, and input / output (IO) cumulative stall time. In this embodiment, a user-space PSI synchronization agent process is executed. For example, the PSI file corresponding to the control group is read every 5ms. As an optional embodiment, the PSI file includes, but is not limited to, CPU files, memory files, and input / output files. The corresponding read locations are / sys / fs / cgroup / ... / cpu.pressure, / sys / fs / cgroup / ... / memory.pressure, and / sys / fs / cgroup / ... / io.pressure, respectively. The CPU file, memory file, and input / output file (PSI file) are parsed to obtain the cumulative stall time of the corresponding CPU, the cumulative stall time of the memory, and the cumulative stall time of the I / O, i.e., total= <ns>The key is cgroup_id, and the cumulative stall time is written to the BPF Map psi_snapshots, which is the second BPF Map.

[0029] S103. Based on the entry probe in the pre-set request probe, obtain the tracking identifier, entry pressure stagnation information snapshot, and entry time, and write them into the pre-set third data packet filter mapping file; In this embodiment, as an optional implementation, the request probe includes, but is not limited to: an extended Berkeley packet filter request probe, which achieves non-intrusive monitoring by pre-implanting eBPF probes in key Linux kernel functions, such as the file I / O functions vfs_read and vfs_write. The extended Berkeley packet filter request probe includes an eBPF entry probe and an eBPF exit probe. As an optional implementation, the eBPF entry probe is an uprobe used for dynamic probing in user-space programs.

[0030] In this embodiment, as an optional embodiment, based on the entry probe in the request probe, the tracking identifier, entry pressure stagnation information snapshot, and entry time are obtained, including: Call the bpf_get_current_pid_tgid function to obtain the identifier of the running thread; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the obtained current control group identifier, query the second data packet filter mapping file to obtain a snapshot of the inlet pressure stagnation information corresponding to the current control group identifier; Record the current entry time; Write the obtained snapshot of the inlet pressure stagnation information, the current inlet time, the tracking identifier, and the current control group identifier into the third data packet filter mapping file.

[0031] In this embodiment, the bpf_get_current_pid_tgid function is used to obtain the current thread identifier and process identifier (pid_tgid, Thread Group ID). As an optional embodiment, the third packet filter mapping file is a request context mapping file, i.e., a BPF Map req_ctx_map, with the key pid_tgid.

[0032] In this embodiment, the corresponding program code segment is as follows: SEC("uprobe / http_handler") int enter_request(struct pt_regs *ctx) { u64 pid_tgid = bpf_get_current_pid_tgid(); u32 tid = (u32)pid_tgid; u64 cgid = bpf_get_current_cgroup_id(); / / Get the trace_id (from user space) u64 *trace_id = bpf_map_lookup_elem(&trace_map,&tid); / / Get PSI snapshot struct psi_snapshot *snap = bpf_map_lookup_elem(&psi_snapshots,&cgid); / / Build Context struct req_ctx ctx = { .trace_id = trace_id ? *trace_id : 0, .start_ns = bpf_ktime_get_ns(), .entry_psi = snap ? *snap : (struct psi_snapshot){0} }; bpf_map_update_elem(&req_ctx_map,&pid_tgid,&ctx, BPF_ANY); return 0; } S104. Based on the exit probe in the request probe, obtain the tracking identifier, exit pressure stagnation information snapshot and exit time; based on the tracking identifier, query the third data packet filter mapping file to obtain the entry pressure stagnation information snapshot and entry time corresponding to the tracking identifier. In this embodiment, as an optional embodiment, the exit probe is an eBPF exit probe, and the eBPF exit probe is an uprobe. In this embodiment, as an optional embodiment, the third data packet filter mapping file is queried based on the tracking identifier to obtain the ingress pressure stagnation information snapshot and ingress time corresponding to the tracking identifier, including: Call the bpf_get_current_pid_tgid function to obtain the process ID to which the running thread ID belongs; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the acquired tracking identifier and current control group identifier, query the third data packet filter mapping file to obtain the inlet pressure stagnation information snapshot and inlet time.

[0033] In this embodiment, the eBPF outlet probe is used to query and obtain a snapshot of the outlet pressure stagnation information and the outlet time.

[0034] S105. Based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time, obtain the pressure stagnation information change value and application delay. Based on the pressure stagnation information change value, application delay, and a preset root cause threshold, determine the root cause.

[0035] In this embodiment, as an optional embodiment, based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time, the pressure stagnation information change value and application delay are obtained, including: Calculate the difference between the outlet pressure stagnation information snapshot and the inlet pressure stagnation information snapshot to obtain the pressure stagnation information change value; The application delay is obtained by calculating the time difference between the exit time and the entry time.

[0036] In this embodiment, the pressure stagnation information change value (ΔPSI) includes, but is not limited to: CPU stagnation information change value (ΔCPU), memory stagnation information change value (ΔMEM), and input / output stagnation information change value (ΔIO).

[0037] In this embodiment, as an optional embodiment, the root cause is determined based on the pressure stagnation information change value, application delay, and a preset root cause threshold, including: Select the largest value from the CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values; Calculate the ratio of the maximum change in stagnant information to application latency; If the ratio is greater than the preset root cause threshold, the root cause label is determined to be the hardware resource contention corresponding to the maximum change value of the stagnant information. If the ratio is not greater than the preset root cause threshold, the root cause label is determined to be the application's logic.

[0038] In this embodiment, root cause determination is performed. If max(ΔCPU, ΔMEM, ΔIO) / application latency > root cause threshold, it is determined to be resource contention. The root cause type is then labeled according to the hardware resource corresponding to the maximum change value of the stall information. For example, if the maximum change value of the stall information among CPU stall information change value, memory stall information change value, and I / O stall information change value is the CPU stall information change value, and the ratio of this CPU stall information change value to the application latency is greater than the root cause threshold, then the root cause of the application is determined to be CPU resource contention. As an optional embodiment, the root cause threshold is set to 50%, that is, if max(d_cpu, d_mem, d_io) / latency > 0.5, a JSON file is generated.

[0039] In this embodiment, the application latency is obtained based on the current time and the entry probe time.

[0040] In this embodiment, the corresponding program code segment is as follows: SEC("uprobe / http_handler_return") int exit_request(struct pt_regs *ctx) { u64 pid_tgid = bpf_get_current_pid_tgid(); struct req_ctx *entry = bpf_map_lookup_elem(&req_ctx_map,&pid_tgid); if (!entry) return 0; u64 cgid = bpf_get_current_cgroup_id(); struct psi_snapshot *exit_snap = bpf_map_lookup_elem(&psi_snapshots,&cgid); if (!exit_snap) goto cleanup; / / Calculate ΔPSI and application latency u64 d_cpu = exit_snap->cpu_ns - entry->entry_psi.cpu_ns; u64 d_mem = exit_snap->mem_ns - entry->entry_psi.mem_ns; u64 d_io= exit_snap->io_ns- entry->entry_psi.io_ns; u64 latency = bpf_ktime_get_ns() - entry->start_ns; / / Report struct event ev = { .trace_id = entry->trace_id, .latency = latency,.d_cpu, .d_mem, .d_io}; bpf_perf_event_output(ctx,&events, BPF_F_CURRENT_CPU,&ev, sizeof(ev)); cleanup: bpf_map_delete_elem(&req_ctx_map,&pid_tgid); return 0; } { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span.attributes": { "resource.stall.type": "memory", "resource.stall.delta_ns": 150000000, "root_cause.category": "memory_pressure" } } In this embodiment, as an optional implementation, the OpenTelemetry integration interface is used to inject root cause tags into the span attribute of the corresponding tracking identifier, thereby achieving standard ecosystem compatibility. For example, the generated or produced JSON file (root cause tags) can be injected into the Application Performance Monitoring (APM) backend via OpenTelemetry Collector or directly.

[0041] In this embodiment, as an optional implementation, the eBPF outlet probe can be used to calculate the pressure stagnation information change value and application delay, and the root cause can be determined based on the pressure stagnation information change value, application delay, and a preset root cause threshold. As another optional implementation, the eBPF outlet probe can also report the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time, for example, to the root cause determination module. The root cause determination module can then be used to calculate the pressure stagnation information change value and application delay, and the root cause can be determined based on the pressure stagnation information change value, application delay, and a preset root cause threshold.

[0042] In this embodiment, as an optional embodiment, the method further includes: If the "some" field in the pressure stall information indicates that no task has stalled due to resource unavailability, then the read cycle is increased.

[0043] In this embodiment, as another optional embodiment, the method further includes: If the "full" field in the pressure stagnation information indicates that all tasks are stalled simultaneously, then the reading cycle should be reduced.

[0044] In this embodiment, when the pressure stagnation information (outlet pressure stagnation information snapshot) indicates that the task scheduling is in a normal state, the read cycle is reduced to reduce kernel overhead. In this embodiment, the method obtains the tracking identifier configured in the application request initiated by the user, writes the application thread identifier executing the application request and the tracking identifier into a pre-set first data packet filter mapping file, reads the ingress pressure stagnation information snapshot recorded by the target control group configured for the application corresponding to the application request according to a pre-set reading cycle, writes the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a pre-set second data packet filter mapping file, obtains the tracking identifier, ingress pressure stagnation information snapshot, and ingress time based on the ingress probe in the pre-set request probe, and writes them into a pre-set third data packet filter mapping file, obtains the tracking identifier, ingress pressure stagnation information snapshot, and ingress time based on the egress probe in the request probe, queries the third data packet filter mapping file based on the tracking identifier to obtain the ingress pressure stagnation information snapshot and ingress time corresponding to the tracking identifier, obtains the pressure stagnation information change value and application latency based on the ingress pressure stagnation information snapshot, ingress time, egress pressure stagnation information snapshot, ingress time, and egress time, and determines the root cause based on the pressure stagnation information change value, application latency, and a pre-set root cause threshold. It has the following beneficial technical effects: High accuracy: Directly quantifies the stall time caused by resource contention during application request execution, with an error of <5ms; Low overhead: Utilizing eBPF probes, CPU usage is <2%, and PSI sampling frequency is adjustable; Minimal intrusion: Only one line of code is needed to call the BPF Map update (which can be automatically injected by the agent); End-to-end traceability: In this embodiment, the trace identifier (trace_id) comes from the OpenTelemetry standard and is seamlessly integrated into existing APM systems; Strong scenario adaptability: By setting up "single process + dedicated cgroup" optimization for applications, multi-tenant interference can be avoided. It is applicable to performance diagnosis in observable domains, including but not limited to: database services, such as PostgreSQL and Redis, AI inference engines, such as TensorRT and TorchServe, microservices, such as Go / Java independent processes, etc. It can effectively achieve high-precision, non-intrusive, and low-overhead latency root cause localization analysis.

[0045] Figure 2 This is another flowchart illustrating a delayed root cause analysis method for an application provided in an embodiment of the present invention. For example... Figure 2 As shown, the process includes: S201. The application process extracts the trace identifier (trace_id) from the application request. In this embodiment, the application process is the process that handles application requests.

[0046] S202. Write the trace identifier into the trace identifier injector (trace_id injector). S203. Write the tracking identifier and application process identifier (tid) into the first BPF Map; In this embodiment, the trace_id written in the first BPF Map forms a mapping relationship with tid. As an optional embodiment, the first BPF Map is trace_map.

[0047] S204, Periodically synchronize PSI to the second BPF Map; In this embodiment, a PSI synchronization agent is used to synchronize the PSI according to a pre-set period, write psi-snapshots, and establish a mapping relationship between the control group identifier and the PSI.

[0048] S205, Using the entry probe, read the tracking identifier from the first BPF Map, and read the PSI snapshot from the second BPF Map; S206. Save the read tracking identifier and PSI snapshot to the third BPF Map; In this embodiment, the entry probe writes the PSI snapshot, the current entry time, and the trace_id into the req_ctx_map file.

[0049] S207. Use the exit probe to read the PSI snapshot from the second BPF Map, and read the req_ctx_map file from the third BPF Map; In this embodiment, the exit probe is an exit uprobe probe, and the req_ctx_map file includes the application process identifier and PSI snapshot.

[0050] S208. Generate a reporting event using the exit probe and send it to the event reporting device; In this embodiment, the event reporter is events (perf buffer).

[0051] S209. The event reporter generates a consumption event and reports it to the root cause determination module. In this embodiment, as an optional embodiment, the consumption event includes an inlet pressure stagnation information snapshot, an inlet time, an outlet pressure stagnation information snapshot, an inlet time, and an outlet time.

[0052] S210, the root cause determination module reports the data of events exceeding the threshold to the OpenTelemetry Collector or APM backend.

[0053] In this embodiment, as an optional implementation, the over-threshold event data is a resource contention label.

[0054] Figure 3 This is another schematic flowchart illustrating a delayed root cause analysis method provided in an embodiment of the present invention. For example... Figure 3 As shown, the process includes: S301, The user initiates an HTTP request to the application process; S302. The application process extracts the trace identifier from the HTTP request and sends it to the trace_id injector. S303, the trace_id injector obtains the application process identifier; The S304 trace_id injector writes the application process identifier and trace identifier into the trace map. S305, the trace_id injector triggers the eBPF entry probe; S306 and eBPF entry probes query and obtain the tracing identifier and application process identifier, and send them to TraceMap; S307, eBPF ingress probe queries to obtain control group identifier and ingress PSI snapshot, and sends them to snapshot container (PsiSnapshots). S308, the eBPF entry probe sends the process identifier, entry PSI snapshot, current entry time, and trace_id to the mapper (req_ctx_map). S309, The application process executes business logic; In this embodiment, the business logic includes, but is not limited to: SQL, reasoning, and calculation.

[0055] S310, The application process sends a probe return to the eBPF exit probe, triggering the process. S311, eBPF exit probe obtains process identifier and sends it to req_ctx_map; S312 and eBPF exit probes acquire control group identifiers and exit PSI snapshots, and send them to PsiSnapshots; The S313 and eBPF outlet probes calculate pressure stagnation information changes based on outlet PSI snapshots and inlet PSI snapshots, and calculate application delays based on the current outlet time and current inlet time. S314, eBPF outlet probe outputs tracking identifier, pressure stagnation information change value and application delay to event reporter; S315. The root cause determination module reads events from the event reporter; In this embodiment, events include: tracking identifiers, pressure stagnation information change values, and application delays.

[0056] S316. The root cause determination module determines the existence of resource contention pressure based on the read events; S317. The root cause determination module injects the span attribute into the OpenTelemetry or APM backend.

[0057] Based on the same inventive concept, such as Figure 4 As shown, this embodiment of the invention also provides an application of delayed root cause analysis device, the device comprising: The first mapping file acquisition module 401 is used to acquire the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; In this embodiment, as an optional embodiment, the first mapping file acquisition module 401 is specifically used for: The user-space tracking identifier injector is used to extract the configured tracking identifier at the entry point of the application request initiated by the user. Obtain the application thread identifier that executes the application request, and write it into the first data packet filter mapping file using the application thread identifier and the tracking identifier as the corresponding key values.

[0058] The second mapping file acquisition module 402 is used to read the ingress pressure stagnation information snapshot recorded by the target control group of the application configuration corresponding to the application request according to a preset reading cycle, and write the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a preset second data packet filter mapping file. In this embodiment, a control group is pre-configured for the application. As an optional embodiment, a dedicated control group is configured for each application. As another optional embodiment, a control group is configured for multiple applications. This embodiment does not limit this.

[0059] The third mapping file acquisition module 403 is used to acquire the tracking identifier, entry pressure stagnation information snapshot, and entry time based on the entry probe in the pre-set request probe, and write them into the pre-set third data packet filter mapping file; In this embodiment, as an optional embodiment, the third mapping file acquisition module 403 is specifically used for: Call the bpf_get_current_pid_tgid function to obtain the identifier of the running thread; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the obtained current control group identifier, query the second data packet filter mapping file to obtain a snapshot of the inlet pressure stagnation information corresponding to the current control group identifier; Record the current entry time; Write the obtained snapshot of the inlet pressure stagnation information, the current inlet time, the tracking identifier, and the current control group identifier into the third data packet filter mapping file.

[0060] The parameter acquisition module 404 is used to acquire the tracking identifier, the snapshot of the outlet pressure stagnation information and the outlet time based on the outlet probe in the request probe, and to query the third data packet filter mapping file based on the tracking identifier to acquire the snapshot of the inlet pressure stagnation information and the inlet time corresponding to the tracking identifier. In this embodiment, as an optional embodiment, the parameter acquisition module 404 is specifically used for: Call the bpf_get_current_pid_tgid function to obtain the process ID to which the running thread ID belongs; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the acquired tracking identifier and current control group identifier, query the third data packet filter mapping file to obtain the inlet pressure stagnation information snapshot and inlet time.

[0061] The delay root cause determination module 405 is used to obtain the pressure stagnation information change value and application delay based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time and outlet time, and determine the root cause based on the pressure stagnation information change value, application delay and a preset root cause threshold.

[0062] In this embodiment, as an optional embodiment, the delay root cause determination module 405 is specifically used for: Calculate the difference between the outlet pressure stagnation information snapshot and the inlet pressure stagnation information snapshot to obtain the pressure stagnation information change value; The application delay is obtained by calculating the time difference between the exit time and the entry time.

[0063] In this embodiment, as another optional embodiment, the pressure stagnation information change value includes: CPU stagnation information change value, memory stagnation information change value, and input / output stagnation information change value. The delay root cause determination module 405 is further used for: Select the largest value from the CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values; Calculate the ratio of the maximum change in stagnant information to application latency; If the ratio is greater than the preset root cause threshold, the root cause label is determined to be the hardware resource contention corresponding to the maximum change value of the stagnant information. If the ratio is not greater than the preset root cause threshold, the root cause label is determined to be the application's logic.

[0064] In this embodiment, as an optional embodiment, the device further includes: The read cycle adjustment module (not shown in the figure) is used to increase the read cycle if the "some" field in the snapshot of the outlet pressure stagnation information indicates that no task is stalled due to resource unavailability. If the full field in the snapshot of export pressure stagnation information indicates that all tasks are stagnant at the same time, then the read cycle is reduced.

[0065] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the delay root cause analysis method of any of the above possible implementations.

[0066] Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0067] Based on the same inventive concept, see [link to inventive concept] Figure 5 This invention also provides an electronic device, including a memory 101 (e.g., non-volatile memory), a processor 102, and a computer program stored on the memory 101 and executable on the processor 102. When the processor 102 executes the program, it implements the steps of the delay root cause analysis method for any of the above possible implementations, which can be equivalent to the delay root cause analysis device for the aforementioned application. Of course, the processor can also be used to process other data or perform calculations. This electronic device can be a PC, server, terminal, or other similar device.

[0068] like Figure 5 As shown, the electronic device may also include: memory 103, network interface 104, and internal bus 105. In addition to these components, other hardware may also be included, which will not be described in detail here.

[0069] It should be noted that the delay root cause analysis device of the above application can be implemented by software. As a logical device, it is formed by the processor 102 of the electronic device in which it is located reading the computer program instructions stored in the non-volatile memory into the memory 103 for execution.

[0070] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0071] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by special-purpose logic circuitry—such as FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), and the device can also be implemented as special-purpose logic circuitry.

[0072] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0073] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0074] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0075] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0076] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0078] The above are merely specific embodiments of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / ns>

Claims

1. A method for analyzing the delayed root causes of application, characterized in that, include: Obtain the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; According to a pre-set reading cycle, read the ingress pressure stagnation information snapshot recorded by the target control group configured for the application request corresponding to the application request, and write the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a pre-set second data packet filter mapping file; Based on the ingress probe in the pre-set request probe, obtain the tracking identifier, ingress pressure stagnation information snapshot, and ingress time, and write them into the pre-set third packet filter mapping file; Based on the exit probe in the request probe, obtain the tracking identifier, exit pressure stagnation information snapshot and exit time; based on the tracking identifier, query the third data packet filter mapping file to obtain the entry pressure stagnation information snapshot and entry time corresponding to the tracking identifier. Based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time, the pressure stagnation information change value and application delay are obtained. Based on the pressure stagnation information change value, application delay, and a pre-set root cause threshold, the root cause is determined.

2. The method for delayed root cause analysis according to claim 1, characterized in that, The step of obtaining the tracking identifier configured in the application request initiated by the user, and writing the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file includes: The user-space tracking identifier injector is used to extract the configured tracking identifier at the entry point of the application request initiated by the user. Obtain the application thread identifier that executes the application request, and write it into the first data packet filter mapping file using the application thread identifier and the tracking identifier as the corresponding key values.

3. The method for delayed root cause analysis according to claim 1, characterized in that, The acquisition of the tracking identifier, inlet pressure stagnation information snapshot, and inlet time based on the inlet probe in the request probe includes: Call the bpf_get_current_pid_tgid function to obtain the identifier of the running thread; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the obtained current control group identifier, query the second data packet filter mapping file to obtain a snapshot of the inlet pressure stagnation information corresponding to the current control group identifier; Record the current entry time; Write the obtained snapshot of the inlet pressure stagnation information, the current inlet time, the tracking identifier, and the current control group identifier into the third data packet filter mapping file.

4. The method for delayed root cause analysis according to claim 1, characterized in that, The step of querying the third data packet filter mapping file based on the tracking identifier to obtain the snapshot of the entry pressure stagnation information and the entry time corresponding to the tracking identifier includes: Call the bpf_get_current_pid_tgid function to obtain the process ID to which the running thread ID belongs; Based on the obtained thread identifier, query the first data packet filter mapping file to obtain the tracking identifier corresponding to the thread identifier; Call the bpf_get_current_cgroup_id function to obtain the current control group identifier corresponding to the running thread identifier; Based on the acquired tracking identifier and current control group identifier, query the third data packet filter mapping file to obtain the inlet pressure stagnation information snapshot and inlet time.

5. The method for delayed root cause analysis of the application according to any one of claims 1 to 4, characterized in that, The process of obtaining pressure stagnation information change values ​​and application delays based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time, and outlet time includes: Calculate the difference between the outlet pressure stagnation information snapshot and the inlet pressure stagnation information snapshot to obtain the pressure stagnation information change value; The application delay is obtained by calculating the time difference between the exit time and the entry time.

6. The method for delayed root cause analysis according to claim 5, characterized in that, The stress stagnation information change values ​​include: CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values. The determination of the root cause based on the stress stagnation information change values, application latency, and a pre-set root cause threshold includes: Select the largest value from the CPU stagnation information change values, memory stagnation information change values, and input / output stagnation information change values; Calculate the ratio of the maximum change in stagnant information to application latency; If the ratio is greater than the preset root cause threshold, the root cause label is determined to be the hardware resource contention corresponding to the maximum change value of the stagnant information. If the ratio is not greater than the preset root cause threshold, the root cause label is determined to be the application's logic.

7. The method for delayed root cause analysis of the application according to any one of claims 1 to 4, characterized in that, The method further includes: If the "some" field in the snapshot of export pressure stall information indicates that no task is stalled due to resource unavailability, then the read cycle is increased. If the full field in the snapshot of export pressure stagnation information indicates that all tasks are stagnant at the same time, then the read cycle is reduced.

8. A device for analyzing the delayed root causes of an application, characterized in that, The application's delay root cause analysis device includes: The first mapping file acquisition module is used to acquire the tracking identifier configured in the application request initiated by the user, and write the application thread identifier that executes the application request and the tracking identifier into a pre-set first data packet filter mapping file; The second mapping file acquisition module is used to read the ingress pressure stagnation information snapshot recorded by the target control group of the application configuration corresponding to the application request according to a preset reading cycle, and write the ingress pressure stagnation information snapshot and the control group identifier of the target control group into a preset second data packet filter mapping file. The third mapping file acquisition module is used to acquire the tracking identifier, entry pressure stagnation information snapshot, and entry time based on the entry probe in the pre-set request probe, and write them into the pre-set third data packet filter mapping file; The parameter acquisition module is used to acquire the tracking identifier, the snapshot of the outlet pressure stagnation information, and the outlet time based on the outlet probe in the request probe, and to query the third data packet filter mapping file based on the tracking identifier to acquire the snapshot of the inlet pressure stagnation information and the inlet time corresponding to the tracking identifier. The delayed root cause determination module is used to obtain the pressure stagnation information change value and application delay based on the inlet pressure stagnation information snapshot, inlet time, outlet pressure stagnation information snapshot, inlet time and outlet time, and determine the root cause based on the pressure stagnation information change value, application delay and a preset root cause threshold.

9. A storage medium, characterized in that, A program or instruction is stored on a storage medium, and the program or instruction is executed by a processor to implement the steps of the delay root cause analysis method of the application as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the delay root cause analysis method of the application according to any one of claims 1 to 7.

Images