Non-intrusive link tracking method and system based on eBPF

Abstract

The invention belongs to the technical field of computer software and network, and discloses an eBPF-based non-intrusive link tracking method, which comprises the following steps of: acquiring network traffic and system call data of each micro-service split by a target application system based on an eBPF technology, and constructing Span data containing request and response information; and carrying out association integration on the Span data by adopting a link tracking algorithm according to the intrinsic characteristics of the target application system to generate distributed call chain Trace data. According to the invention, full-link and zero-intrusive link tracking of the micro-service application from the service layer to the infrastructure layer in the cloud native environment can be realized.

Description

Technical Field

[0001] This invention belongs to the field of computer software and network technology, specifically relating to a non-intrusive link tracing method and system based on eBPF, which is particularly suitable for high-concurrency, multi-language, and heterogeneous microservice applications in cloud-native environments, and is used to achieve end-to-end performance monitoring and fault diagnosis. Background Technology

[0002] As the scale and complexity of internet applications continue to grow, traditional monolithic architectures are no longer able to meet the increasingly complex and dynamic business needs. Microservice architecture, as a widely favored software development paradigm and organizational method, has emerged. It breaks down a single application into a set of small, independent microservices, each running in its own process, and interacting with each other through lightweight communication mechanisms (such as HTTP or RPC).

[0003] With the widespread adoption of cloud-native and microservice architectures, application systems are being broken down into numerous independently deployed microservices that communicate over a network. To ensure system stability and efficient fault location, distributed tracing technology has become crucial, enabling the rapid acquisition of performance data from microservices, such as request latency. Traditional tracing tools often employ an "intrusive" approach, requiring the embedding of tracing code into the application code to generate and transmit the trace context, including trace_id and parent_span_id.

[0004] However, this invasive tracking technology has the following significant drawbacks.

[0005] 1) Incomplete data coverage in the call chain: In a typical microservice call chain, requests pass through middleware such as API gateways, message queues (e.g., Kafka), and databases (e.g., MySQL). Since the source code of these middleware cannot be modified, intrusive probes can usually only collect the client's request span, but cannot obtain complete information about the server's processing (e.g., SQL execution time, Kafka consumption latency), resulting in "breakpoints" in the call chain and making end-to-end full-chain tracing impossible.

[0006] 2) Limitations of Intrusive Probes: Traditional solutions (such as Zipkin and Skywalking) require embedding the SDK into the application code. This not only increases development and maintenance costs, but also necessitates maintaining multiple probes for microservices written in different languages ​​(Java, Go, Python, etc.). More importantly, the introduction of probe code may have unpredictable impacts on application performance and stability.

[0007] 3) Risk of lost trace context: When making cross-service calls, if the middleware or a certain service fails to pass trace_id and other trace context correctly, the local call chain will be broken, making subsequent chain construction extremely difficult.

[0008] Therefore, the industry urgently needs a link tracing technology that can achieve full-link, zero-intrusion operation to solve the above problems. Summary of the Invention

[0009] The purpose of this invention is to provide a non-intrusive link tracing method and system based on eBPF, aiming to overcome the shortcomings of existing technologies and achieve full-link, zero-intrusion link tracing of microservice applications in cloud-native environments from the service layer to the infrastructure layer.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A non-intrusive link tracing method based on eBPF includes the following steps: Based on eBPF technology, network traffic and system call data of each microservice after the target application system is split are collected, and Span data containing request and response information is constructed. The Span data is associated and integrated based on the inherent characteristics of the target application system using a link tracing algorithm to generate distributed call chain Trace data.

[0011] Preferably, the intrinsic characteristics include API dependency information, request execution information, and request semantic information.

[0012] Preferably, the method for obtaining the dependency information of the API is as follows: 11) Static program analysis techniques are used to parse the source code of the microservices and construct the abstract syntax tree, control flow graph and call graph of the microservices to identify direct calls, indirect calls and potential dynamic call relationships between functions; 12) Extract the methods and parameters of the RPC service, or the path, request / response structure and parameter dependencies of the HTTP API, from the interface definition document of the microservice; 13) Cross-integrate the dependency information obtained in steps 11) and 12) to construct a directed API dependency graph, providing constraints for association decisions in the link tracing process.

[0013] Preferably, the method for obtaining the execution information of the request is as follows: First, parse the source code of the microservice, identify its concurrent execution model, and output a concurrent call graph; if the microservice is determined to be a multi-threaded reuse model microservice based on the concurrent execution model, then determine whether its request processing logic crosses different operating system threads during execution; if the microservice is determined to be a single-threaded serial model microservice based on the concurrent execution model, then confirm that all its operations are executed sequentially in a single operating system thread; then, use the concurrent call graph to identify all API sets that do not cross thread contexts in a complete request processing flow, as the execution information of the corresponding request.

[0014] Preferably, the method for obtaining the semantic information of the request is as follows: First, the information entropy algorithm is used to identify and extract the set key business fields from the request message body of the Span data; then, the co-occurrence frequency of the key business fields in the upstream and downstream API calls is counted, and the semantic similarity between different key business fields is calculated; then, by combining the co-occurrence frequency and semantic similarity between the key business fields, a request semantic correlation matrix is ​​constructed, which is used to evaluate the consistency of the business context between upstream and downstream Spans during online tracing.

[0015] Preferably, a hierarchical decision-making mechanism is used to select the optimal link tracing algorithm; the priority order of the hierarchical decision-making mechanism is: tracing based on execution information > tracing based on message content > tracing based on message latency; the tracing method based on execution information is: if the upstream API and downstream API to be associated are in the same execution context, then they are directly associated; the tracing method based on message content is: if there are high-information-content fields in the message bodies of the upstream and downstream spans and the message values ​​match, then they are associated; the tracing method based on message latency is: using a Gaussian mixture model to select the candidate span with the highest temporal likelihood to establish the association relationship.

[0016] Preferably, the message latency-based tracing method is as follows: First, based on the happens-before rule, according to the end timestamp of the upstream Span and the start timestamp of the downstream Span, a set of all downstream Spans that satisfy the time sequence constraint are identified for each upstream Span to be associated; then, combined with the service call topology and logical order between operations defined by the pre-built API dependency graph, the downstream Span set is filtered to determine a set of candidate associations that are most likely in terms of time and call logical order, where each element in the candidate association set corresponds to a downstream service and its sub-Spans; then, for each Span in the candidate association set... n collects message latency data and generates a Gaussian mixture model based on the message latency data. The Gaussian mixture model is then fitted with parameters by iteratively executing the expectation-maximization algorithm to obtain the probability distribution of different latency modes. The likelihood probability of each candidate association is calculated based on the fitted Gaussian mixture model. Then, the candidate association with the highest likelihood probability value is selected as the final association result. The parent_span_id field is then assigned to each Span to establish the call hierarchy relationship, and the same trace_id is uniformly assigned to all Spans under the same distributed transaction context, thereby realizing the reconstruction of the call chain across services and processes and obtaining the distributed call chain Trace of the microservice.

[0017] Preferably, the method for collecting network traffic and system call data of each microservice after the target application system is split based on eBPF technology is as follows: when a network data packet flows through a pre-set eBPF hook in the kernel, the corresponding eBPF program is triggered to execute, and the data packet is captured and parsed in real time to generate structured kernel event data and stored in the eBPFMaps memory area; the method for constructing Span data containing request and response information is as follows: the user-space program first reads the kernel event data from the eBPFMaps memory area, parses its payload content, and constructs a Span structure containing requests and responses, covering timestamps and message content; then, according to the mapping relationship between TCP 5-tuples and sockets, it associates kernel events generated by different eBPF hooks that belong to the same network connection; then, combined with the mapping relationship between the process PID to which the kernel event belongs and the container ID in the container cgroup path, it determines the container name and service name corresponding to the Span data.

[0018] A non-intrusive link tracing system based on eBPF, characterized in that it includes an eBPF program module, an Agent module, a Tracer module, and a Server module; The eBPF program module is used to load and run eBPF bytecode programs in kernel mode. It uses eBPF hooks set in the kernel network stack and system call entry points. When network packets flow through the pre-set eBPF hooks in the kernel, the corresponding eBPF program is triggered to execute, and the packets are captured and parsed in real time to generate structured kernel event data and store it in the eBPF Maps memory area. The Agent module is used to compile the eBPF program source code into eBPF bytecode and load it into the kernel, and to read kernel event data generated by the kernel from the eBPF Maps memory area and process it into structured Span data. The Server module is used to obtain Span data from the user-mode processing module and is responsible for querying Trace data required by the Web module. The Tracer module is used to associate and integrate scattered Span data to generate distributed call chain Trace data.

[0019] A computing device, characterized in that it comprises: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above.

[0020] Compared with the prior art, the advantages of the present invention are as follows: First, a non-intrusive network traffic collection mechanism is adopted to directly capture raw network data packets at the transport and application layers, rather than relying on log information actively output by the target service. This method avoids inherent defects such as inconsistent log formats and missing key context fields, ensuring that the link tracing data has complete temporal sequence, end-to-end coverage, and is independent of programming languages. Second, it completely abandons the bytecode instrumentation implementation path based on Java Agents, without introducing any runtime class loading hijacking, method enhancement, or strong JVM parameter constraints, and naturally supports multiple programming language stacks (including but not limited to Go, Python, Node.js, Rust, and C++). This method decouples the tracing process from the application process, eliminating the coupling dependency on the target application's language ecosystem, runtime version, and containerization configuration, and avoiding performance fluctuations and operational risks caused by instrumentation, significantly improving the universality and stability of the tracing system. Third, the technological focus is on the real-time, non-intrusive, and adaptive construction of distributed call chains, rather than performing retrospective analysis or anomaly pattern recognition on the generated chain data. Attached Figure Description

[0021] Figure 1 This is a diagram illustrating the implementation architecture of the system of the present invention.

[0022] Figure 2This is a diagram showing the implementation architecture of the Tracer core module.

[0023] Figure 3 A flowchart for Tracer's hierarchical decision-making mechanism.

[0024] Figure 4 This is a flowchart of the Execution Information-Based Tracing (EbT) method.

[0025] Figure 5 This is a flowchart of the Message Content-Based Tracing (CbT) method.

[0026] Figure 6 This is a flowchart of the Message Delay-Based Tracing (DbT) method. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0028] An embodiment of the present invention provides a non-intrusive link tracing method based on eBPF, characterized by comprising the following steps: Based on eBPF technology, network traffic and system call data of each microservice of the target application system are collected to construct Span data containing request and response information. The inherent characteristics of the target application system are pre-analyzed through a preprocessing procedure, including API dependency information, request execution information, and request semantic information. Based on the aforementioned characteristics, the online tracing program selects the optimal link tracing algorithm using a hierarchical decision-making mechanism, including tracing based on execution information, tracing based on request content, and tracing based on request latency. It then infers the correlation relationships of the Span data and generates complete distributed call chain trace data.

[0029] In this embodiment, the generated distributed call chain trace data is output to the backend storage system Clickhouse, or other storage systems that conform to the OpenTelemetry standard, through the OpenTelemetry standardized interface. The backend system uses the distributed call chain trace data to construct a service topology diagram and a trace tree diagram for visualization and querying.

[0030] In an optional embodiment of the present invention, the step of collecting network traffic, network connections, thread descriptors, and system call data of microservice applications based on eBPF technology includes: Set eBPF hooks at the network stack and system call entry points in the Linux kernel. These hooks involve system calls such as sendmsg / recvmsg, write / read, and listen / accept. The eBPF program captures kernel events of network packet reading and writing, obtains the source / destination IP address, port, protocol, payload content and timestamp of the request, and constructs a data structure (Event structure) to describe the kernel event. The eBPF program captures kernel events of TCP connection establishment and termination, obtains the source / destination IP address and port 5-tuple of the TCP connection, and stores the mapping relationship between the 5-tuple and socket descriptor (FD) in eBPF Maps. The eBPF program captures kernel events such as process creation, thread switching, and context switching, obtains process PID and thread TID information, and uses this information as key fields of Event data. The captured raw data is preprocessed in kernel space and converted into structured event data with timestamps. The event data is then passed to the user space program through eBPF Maps. The user-space program reads Event data from eBPF Maps, parses its payload content, and constructs a Span structure containing requests and responses, including timestamps and message content. Based on the mapping relationship between TCP 5-tuples and sockets, the user-space program associates Event events generated by different eBPF hooks that belong to the same network connection. Note that the eBPF program captures kernel events on any process, so it needs to be aware of the containers and services corresponding to the process; the user-space program combines the process PID with the container ID mapping relationship in the container cgroups (Control Groups) path to determine the container name and service name corresponding to the Span data, and writes them as attributes into the Span structure.

[0031] In an optional embodiment of the present invention, the dependency information of the API is obtained in the following manner: Using static program analysis tools, Large Language Models (LLM), or other intelligent program analysis tools, analyze the source code of microservices in languages ​​such as Java and Go, as well as interface definition documents in formats such as Protobuf, Swagger, and OpenAPI, to infer the call order and dependencies between APIs within the service, thereby constructing an API dependency graph; specifically including: First, for the source code of microservices, static program analysis technology is used to parse its abstract syntax tree (AST) and construct the control flow graph (CFG) and call graph to identify direct calls, indirect calls and potential dynamic call relationships between functions; Secondly, for the interface definition documents of microservices, parse the Protobuf description file to extract the methods and parameters of the RPC service, or parse the Swagger and OpenAPI description files to obtain the HTTP API path, request / response structure and parameter dependencies; Finally, the dependency information obtained from the source code and interface documentation is cross-integrated to construct a directed API dependency graph, which provides constraints for association decisions during the tracing process.

[0032] In an optional embodiment of the present invention, the execution information of the request is obtained in the following manner: Static analysis tools are used to identify the concurrent execution models of microservices, including single-threaded serial models and multi-threaded multiplexing models; specifically including: First, in the preprocessing stage, the source code of the microservice is analyzed using program analysis tools. Based on the programming language, framework and asynchronous programming paradigm used, the concurrent execution model to which it belongs is automatically identified, and the output is a concurrent call graph. Then, for microservices identified as multi-threaded reuse models, determine whether their request processing logic crosses different operating system threads during execution; for operations that do not cross threads, we can use the equivalence of thread IDs to associate the operation's span; for operations that cross threads, we need to use the other two methods for tracing. Finally, a concurrent call graph is used to identify all upstream and downstream API sets that do not cross thread contexts in a complete request processing flow. The call graph describes each microservice individually. For each microservice, its upstream API is the API that other client services request from this service, and its downstream API is the API that this service requests from other services as a client. For a certain microservice, its upstream span is the span that other client services request from this service, and its downstream API is the span that this service requests from other services as a client.

[0033] In an optional embodiment of the present invention, the semantic information of the request is obtained in the following manner: In the preprocessing stage, in-depth analysis is performed on historically collected Span data to extract business semantics from the message bodies. First, the information entropy algorithm from the field of natural language processing is used to analyze a large number of request message bodies (such as HTTP Body, JSON, Protobuf) to identify and extract key business fields with high information content, such as user_id, order_id, and transaction_id. Then, the co-occurrence frequency of these key business fields in upstream and downstream API calls is calculated. For example, the frequency with which a user_id value appears in a downstream API request when a certain user_id value appears in an upstream API request is also calculated. Simultaneously, the semantic similarity between different field names was analyzed. Using edit distance or word vector models, it was determined that user_id has a high semantic relevance with fields such as customer_id and client_id. Finally, by combining co-occurrence frequency and name similarity, a request semantic association matrix is ​​constructed. This matrix not only records direct matches such as user_id=user_001 and user_id=user_001, but also quantifies indirect but strongly correlated matches such as user_id=user_001 and customer_id=user_001. This matrix serves as the core knowledge base of the CbT algorithm, used to evaluate the consistency of business context between upstream and downstream spans during online tracing. This ensures that even when trace_id is lost, high-confidence span associations can still be performed based on strong business relationships.

[0034] In an optional embodiment of the present invention, the optimal link tracing algorithm is selected using a hierarchical decision-making mechanism, wherein the priority order of the hierarchical decision-making mechanism is: Execution-based Tracing (EbT) > Content-based Tracing (CbT) > Delay-based Tracing (DbT). Specific decision-making mechanism: First, try execution-based tracing (EbT): if the upstream API and downstream API to be associated are in the same execution context (same thread), then associate them directly; If EbT cannot determine the association, then attempt to trace based on message content (CbT): if there are high information content fields in the message bodies of the upstream and downstream Spans and the message values ​​can be matched, then an association is established; If neither of the above two methods can determine the association, then message delay-based tracing (DbT) is enabled: combining the time window and delay distribution model of the API dependency graph, using a Gaussian mixture model (GMM), a time series statistical model, for fitting, and selecting the candidate span with the highest temporal likelihood to establish the association.

[0035] In an optional embodiment of the present invention, the message latency-based tracing is characterized by comprising: First, based on the happens-before rule, according to the end timestamp of the upstream span and the start timestamp of the downstream span, a set of all downstream spans that satisfy the time sequence constraint is identified for each upstream span to be associated. Then, by combining the service call topology and logical order of operations defined by the pre-built API dependency graph, the candidate downstream spans are filtered to determine a set of candidate associations that are most likely in terms of time and call order. Each element in the candidate association set corresponds to a possible downstream service and downstream span. This likelihood estimation step is independent of the model and parameters. Next, message latency data is collected for each Span in the candidate association set, and a Gaussian Mixture Model (GMM) is generated based on the one-dimensional message latency data. The Gaussian Mixture Model is then fitted with parameters by iteratively executing the Expectation Maximization (EM) algorithm to obtain the probability distribution of different latency modes. Subsequently, the candidate sorter calculates the likelihood probability of each candidate association based on the fitted GMM model and sorts them from high to low according to the likelihood value. Finally, the candidate association with the highest likelihood probability score is selected as the inference result of the Span association.

[0036] In an optional embodiment of the present invention, the Trace assembly process is characterized by comprising: The core process is to assign the parent_span_id field to each Span to establish the call hierarchy, and to uniformly assign the same trace_id to all Spans under the same distributed transaction context, thereby realizing the reconstruction of the call chain across services and processes and obtaining the distributed call chain Trace of the microservice. First, a dynamic merging strategy similar to Union-Find is adopted to resolve the call direction based on the caller / callee identifiers explicitly carried between Spans, and to convert the inferred Span association relationship into a directed Span parent-child relationship. Then, an entry service determination rule is introduced. Only when a Root Span that meets the preset conditions (such as a service name of front-end) is identified will the generation of trace_id and the assignment of the entire chain be triggered. Intermediate Spans that do not match the entry conditions are temporarily stored in a memory buffer, and backtracking and completion will be triggered when the subsequent associated Spans arrive, thus balancing the integrity of the chain and the controllability of resources. Finally, the Trace structure serialization was completed in strict accordance with the OpenTelemetry Protocol (OTLP) data model specification to ensure protocol compatibility and efficient integration with mainstream Trace backends (such as Jaeger and Tempo).

[0037] Embodiments of the present invention also provide a non-intrusive link tracing system based on eBPF, characterized in that it includes: an eBPF program module, an Agent module, a Tracer module, a Server module, a storage module, and a Web interface module; wherein: The eBPF program module is used to load and run eBPF bytecode programs in kernel mode. By setting eBPF hooks at the kernel network stack and system call entry points, when network packets flow through the pre-set eBPF hooks in the kernel, the corresponding eBPF program is triggered to execute, and the packets are captured and parsed in real time to generate structured kernel event data and stored in the eBPF Maps memory area. The Agent module is used to compile the eBPF program source code written in C language into eBPF bytecode through the LLVM toolchain, and load it into the kernel through the eBPF Loader component. It is also responsible for reading event data generated by the kernel from eBPF Maps and processing it into structured Span data. The Tracer module is used to pull structured Span data from the Agent module, and infer the relationship between upstream and downstream Spans by using execution-based tracing (EbT), message-content-based tracing (CbT), and message-latency-based tracing (DbT) methods, and assemble complete distributed call chain trace data according to the Span relationship. The Server module is used to pull structured Trace data from the Tracer module and report it to the storage module in OpenTelemetry format. It is also responsible for querying the Trace data required by the Web module. The storage module is used to store trace data and build high-performance indexes for fields such as timestamp and service. The web interface module is used to provide users with visualization of trace data, interactive querying, and real-time monitoring and alarm functions.

[0038] like Figure 1 As shown, the non-intrusive link tracing system based on eBPF of the present invention mainly consists of modules such as eBPF Agent, Tracer, Server, storage and Web interface.

[0039] The eBPF Agent module can be divided into a kernel-mode program responsible for collecting data and a user-mode program responsible for processing data. The kernel-mode Agent program, utilizing the eBPF programming mechanism, can be deployed at the operating system kernel level to collect event data in real time, including network connection events, network request read / write events, and disk file read / write events. The user-mode Agent program identifies and processes the event data and network traffic, distinguishing network connections and service requests from the vast amount of data, and ultimately aggregating them into Span data.

[0040] Supporting multiple operating environments, the eBPF Agent module can be deployed on a Kubernetes Node as a Kubernetes DaemonSet or on a (Virtual) Machine as a Systemd. This allows the Agent to collect and process data from various operating environments, such as Kubernetes systems and (Virtual) Machine systems, as well as from various containers or processes, such as Kubernetes ReplicaSet, Kubernetes StatefulSet, and Host Systemd.

[0041] The kernel-mode eBPF program of the eBPF Agent module is compiled into bytecode using the LLVM suite and loaded into the Linux kernel by the eBPFLoader. The eBPF program sets hooks and attaches itself at critical system calls such as sendmsg / recvmsg, write / read, listen / accept, and at the points of entry and exit from system calls to monitor these calls. When network data I / O events occur, the eBPF program immediately captures the raw network traffic data, extracting the source / destination IP, port, protocol, payload content, and precise timestamp. To minimize the overhead and performance loss of data copying between kernel and user space, the eBPF program parses and filters the raw data packets in kernel space, constructing structured events and storing them in the eBPF Maps structure, thus completing the data transfer from kernel space to user space.

[0042] The eBPF Agent's microservice awareness is achieved through the following mechanism. The eBPF program utilizes eBPF helper functions to efficiently obtain the process ID (PID) of the process that triggered the current event and embeds it as a key field into each generated Event data. The user-space Agent program periodically reads these Events carrying PID information from Maps. During the process of organizing Event data and constructing Spans, the Agent, leveraging its built-in service discovery mechanism, can accurately identify the microservice to which the event belongs based on the PID and bind the Span data to the specific service instance.

[0043] The Tracer tracing module is the core of this invention, and its workflow is as follows: Figure 2 As shown. This module originates from in-depth research on causal relationship judgment methods in traditional distributed tracing, and classifies these methods into three main categories: those based on thread execution information, those based on request semantics, and those based on request time latency. This invention innovatively proposes a hierarchical decision-making mechanism (EbT > CbT > DbT), which does not simply use one of these methods, but organically integrates and prioritizes these three ideas to address the complex challenges of constructing distributed call chains. Its decision-making process is as follows: Figure 3 As shown.

[0044] Execution information-based tracing (EbT) (e.g.) Figure 4 As shown in the diagram): This method is the most accurate tracing approach. It leverages the low-level characteristics of program execution, essentially establishing an "implicit context ID" at the process / thread level. In the thread reuse model described by process-1 (such as a Java Tomcat program), if the processing of a request (upstream Span) and its downstream request (downstream Span) occur under the same thread ID, they can be determined to belong to the same call chain. A similar relationship exists in the single-threaded model described by process-3. However, in the coroutine model described by process-2 (such as a Golang program), since coroutines are lightweight user-space threads, there is no stable relationship between them, unlike threads which have a fixed context switching mechanism. Therefore, EbT cannot handle such cases. The advantage of this method is that the causal relationship is clear, and it almost never produces false associations.

[0045] Content-based tracking (CbT) (e.g.) Figure 5As shown in the diagram, this method, as an enhancement and supplement to EbT, does not directly rely on the ID injected into the code. Instead, it infers causal relationships by analyzing business data in the message body. During the preprocessing stage, the system analyzes historical traffic to learn high-information business fields (such as user_id and order_id) and the correlation matrix between these fields (e.g., user_id and customer_id are highly correlated). During online tracing, for spans that cannot be associated through EbT, the system checks their message bodies (e.g., JSON or Protobuf format). If two candidate spans have the same user_id field value, they are considered to belong to the same call chain with extremely high confidence. This method is particularly suitable for scenarios where trace_id is lost but the business context remains consistent, effectively compensating for the shortcomings of traditional intrusive methods.

[0046] Message latency tracing (DbT) (e.g.) Figure 6 (As shown): This method, as a final fallback strategy, is based on the core idea of ​​causal inference based on timestamps. In the preprocessing stage, the system learns the system's call graph and the order of APIs based on code analysis or historical data. In the online tracing stage, for spans that still cannot be associated, the system generates a set of candidate associations within a reasonable time window, based on the sending time of the upstream request and the receiving time of the downstream request, combined with the topological constraints of the call dependency graph. For example, according to research reports, relying solely on timestamp size may lead to incorrect judgments; this invention, by introducing "order of events" as a strong constraint, eliminates impossible associations. Then, a Gaussian Mixture Model (GMM) is used to model the historical delay distribution, sort the candidate set, and select the association with the best temporal fit and highest likelihood. This method ensures that even in extreme cases, the system can still generate a complete call chain, avoiding the breakage of the distributed call chain.

[0047] The Server module pulls trace data from the Tracer module. This module is designed with a data pipeline conforming to the OpenTelemetry specification, ensuring excellent compatibility and scalability. After receiving raw Span data from the Agent, the Server standardizes it, converting it into the standard OpenTelemetry Protocol (OTLP) format. The processed trace data is then sent to downstream storage modules through this unified data pipeline. This design allows the system to be seamlessly integrated into any observability ecosystem that supports OpenTelemetry, whether using standard components such as Jaeger and OTeL-Collector, or other compatible custom components.

[0048] The storage module, based on Clickhouse, a high-performance, scalable time-series database, is used to persist massive amounts of Span data. This module receives and stores structured Span data forwarded by the Server module. To support efficient querying and analysis, the database has created dedicated indexes on key fields. In particular, high-performance indexes have been created on the two most frequently used query dimensions: timestamp and service name. This optimization enables the system to quickly respond to requests to retrieve historical data by time range or filter specific microservice call chains by service name, providing robust and efficient data support for upper-layer tracing algorithms and visualization queries.

[0049] The web interface module reads the generated trace data from the storage module, providing users with interactive query, analysis, and visualization capabilities. Users can perform complex queries and customize dashboards to monitor key performance indicators through the graphical interface. For distributed call chain data, the tool features a variety of rich display methods, such as trace tree diagrams and flame graphs, helping operations and development personnel intuitively understand the structure and performance bottlenecks of the call chain. Simultaneously, the web interface provides real-time data updates and alerts, ensuring users receive timely feedback on system status and can take action when necessary.

[0050] Through the above implementation methods, the present invention successfully achieves non-intrusive, end-to-end, and high-precision performance monitoring and fault diagnosis for microservice applications.

[0051] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the above-described method. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0052] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A non-intrusive link tracing method based on eBPF, comprising the following steps: Based on eBPF technology, network traffic and system call data of each microservice after the target application system is split are collected, and Span data containing request and response information is constructed. The Span data is associated and integrated based on the inherent characteristics of the target application system using a link tracing algorithm to generate distributed call chain Trace data.

2. The method according to claim 1, characterized in that, The intrinsic characteristics include API dependency information, request execution information, and request semantic information.

3. The method according to claim 2, characterized in that, The method for obtaining the dependency information of the API is as follows: 11) Static program analysis techniques are used to parse the source code of the microservices and construct the abstract syntax tree, control flow graph and call graph of the microservices to identify direct calls, indirect calls and potential dynamic call relationships between functions; 12) Extract the methods and parameters of the RPC service, or the path, request / response structure and parameter dependencies of the HTTP API, from the interface definition document of the microservice; 13) Cross-integrate the dependency information obtained in steps 11) and 12) to construct a directed API dependency graph, providing constraints for association decisions in the link tracing process.

4. The method according to claim 2, characterized in that, The method for obtaining the execution information of the request is as follows: First, parse the source code of the microservice, identify its concurrent execution model, and output the concurrent call graph; if the microservice is determined to be a multi-threaded reuse model microservice based on the concurrent execution model, then determine whether its request processing logic crosses different operating system threads during execution; if the microservice is determined to be a single-threaded serial model microservice based on the concurrent execution model, then confirm that all its operations are executed sequentially in a single operating system thread. Then, the concurrent call graph is used to identify all operations that do not cross thread contexts in a complete request processing flow, which serve as the execution information for the corresponding request.

5. The method according to claim 2, characterized in that, The method for obtaining the semantic information of the request is as follows: First, the information entropy algorithm is used to identify and extract the set key business fields from the request message body of the Span data; then, the co-occurrence frequency of the key business fields in the upstream and downstream API calls is counted, and the semantic similarity between different key business fields is calculated; then, by combining the co-occurrence frequency and semantic similarity between the key business fields, a request semantic correlation matrix is ​​constructed, which is used to evaluate the consistency of the business context between upstream and downstream Spans during online tracing.

6. The method according to claim 1, 2, or 3, characterized in that, A hierarchical decision-making mechanism is adopted to select the optimal link tracing algorithm. The priority order of the hierarchical decision-making mechanism is: tracing based on execution information > tracing based on message content > tracing based on message latency. The tracing method based on execution information is: if the upstream API and downstream API to be associated are in the same execution context, then they are directly associated. The tracing method based on message content is: if there are high-information fields in the message bodies of the upstream and downstream spans and the message values ​​match, then they are associated. The tracing method based on message latency is: using a Gaussian mixture model to select the candidate span with the highest temporal likelihood to establish the association relationship.

7. The method according to claim 6, characterized in that, The message latency-based tracing method is as follows: First, based on the happens-before rule, according to the end timestamp of the upstream span and the start timestamp of the downstream span, a set of all downstream spans that satisfy the time sequence constraint are identified for each upstream span to be associated; then, combined with the service call topology and logical order between operations defined by the pre-built API dependency graph, the downstream span set is filtered to determine a set of candidate associations that are most likely in terms of time and call logical order. Each element in the candidate association set corresponds to a downstream service and its sub-spans; then, for each span in the candidate association set... The system collects message latency data and generates a Gaussian mixture model based on this data. It iteratively executes the expectation-maximization algorithm to fit the parameters of this Gaussian mixture model, thereby obtaining the probability distribution of different latency patterns. The likelihood probability of each candidate association is calculated based on the fitted Gaussian mixture model. Then, the candidate association with the highest likelihood probability is selected as the final association result. Finally, a parent_span_id field is assigned to each Span to establish the call hierarchy, and the same trace_id is uniformly assigned to all Spans under the same distributed transaction context, thus achieving cross-service and cross-process call chain reconstruction and obtaining the distributed call chain trace of the microservice.

8. The method according to claim 1, 2, or 3, characterized in that, The method for collecting network traffic and system call data of each microservice after the target application system is split based on eBPF technology is as follows: When a network data packet flows through a pre-set eBPF hook in the kernel, the corresponding eBPF program is triggered to execute, and the data packet is captured and parsed in real time to generate structured kernel event data and stored in the eBPF Maps memory area; The method for constructing Span data containing request and response information is as follows: The user-space program first reads the kernel event data from the eBPF Maps memory area, parses its payload content, and constructs a Span structure containing requests and responses, covering timestamps and message content; then, according to the mapping relationship between TCP 5-tuples and sockets, it associates kernel events generated by different eBPF hooks that belong to the same network connection; then, combined with the mapping relationship between the process PID to which the kernel event belongs and the container ID in the container cgroup path, it determines the container name and service name corresponding to the Span data.

9. A non-intrusive link tracing system based on eBPF, characterized in that, Includes the eBPF program module, Agent module, Tracer module, and Server module; The eBPF program module is used to load and run eBPF bytecode programs in kernel mode. It uses eBPF hooks set in the kernel network stack and system call entry points. When network packets flow through the pre-set eBPF hooks in the kernel, the corresponding eBPF program is triggered to execute, and the packets are captured and parsed in real time to generate structured kernel event data and store it in the eBPF Maps memory area. The Agent module is used to compile the eBPF program source code into eBPF bytecode and load it into the kernel, and to read kernel event data generated by the kernel from the eBPFMaps memory area and process it into structured Span data. The Server module is used to obtain Span data from the user-mode processing module and is responsible for querying Trace data required by the Web module. The Tracer module is used to associate and integrate scattered Span data to generate distributed call chain Trace data.

10. A computing device, characterized in that, include: A processor, a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method as described in any one of claims 1 to 8.

Images