An application call link tracing method and device and a distributed system
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHAOTONG LIANGFENGTAI INFORMATION TECH CO LTD
- Filing Date
- 2023-01-17
- Publication Date
- 2026-07-21
AI Technical Summary
Existing link tracing technology requires the introduction of components, which increases resource consumption and affects system operating efficiency.
By providing an eBPF program call interface in the kernel, a unique link identifier is generated, and basic information and response information of the request source application and target application are associated. This information is written to a message queue to generate link information, which is then stored and analyzed in a log storage center to achieve full-link tracing.
Reduce resource consumption, improve application processing efficiency, achieve full-link tracing without intrusion, and improve system operating efficiency.
Smart Images
Figure CN116069608B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed system tracing technology, and in particular to an application call tracing method, apparatus and distributed system. Background Technology
[0002] In large-scale distributed systems, there are hundreds or thousands of services developed, and the relationships between them are very complex. Therefore, collecting system call chain data in the load call relationship and analyzing it to identify call chains with bugs or performance problems is beneficial to improving the system.
[0003] There are various methods for distributed system tracing, such as Skywalking, which is a popular and powerful implementation. Its main characteristic is its non-intrusive approach, involving bytecode injection and data transmission to storage devices via HTTP or gRPC. Data processing and analysis are then presented in a web-based visualization. Prior to cloud-native computing, this method was highly efficient and practical, and widely used.
[0004] However, the Skywalking component dependency needs to be introduced into the application, and this proxy component needs to be parasitic on the application service itself to collect data. This means that some of the resources already allocated to the application will be occupied, reducing the available resources of the application itself and thus affecting the running efficiency of the system application. Summary of the Invention
[0005] In order to overcome the above-mentioned technical defects, the purpose of this invention is to provide an application call link tracing method, device and distributed system to solve the problems of existing link tracing requiring the introduction of components, increasing resource consumption and low system operating efficiency.
[0006] This invention discloses an application call chain tracing method, comprising:
[0007] A pre-defined eBPF application call interface is provided to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application.
[0008] A source application requests to invoke a target application and receives a response from the target application;
[0009] The requesting application synchronously triggers an eBPF program in the kernel to generate a unique link identifier, associates the basic information and response information of the requesting application and the target application, and writes it into the message queue to generate link information.
[0010] The log storage center consumes the message queue and processes and stores it in the log storage server, so as to perform link tracing of application calls in the log storage server according to the link identifier.
[0011] Preferably, the step of generating a unique link identifier, associating it with the basic information and response information of the request source application and the target application, and writing it into a message queue to generate link information includes:
[0012] Obtain basic information about the request source application and the target application, and add a unique link identifier;
[0013] Collect network status codes, network response messages, target program response status codes and response messages, and the current system time to obtain response information;
[0014] The response time is calculated based on the collection time of the response information and the collection time of the basic information of the requesting source application.
[0015] After the response time, the basic information and response information are sequentially written into the message queue to generate link information corresponding to the unique link identifier.
[0016] Preferably, the basic information of both the requesting source application and the target application includes at least one of the following:
[0017] Application name;
[0018] method;
[0019] port;
[0020] IP;
[0021] Current system time.
[0022] Preferably, before the requesting source application synchronously triggers the eBPF program in the kernel, the method further includes:
[0023] The GPL license is pre-configured in the kernel, and the GPL-licensed program is loaded before the eBPF program is triggered.
[0024] Preferably, the pre-provided eBPF program calling interface includes:
[0025] Run the command to compile the source file of the eBPF program, and compile the source file of the eBPF program into an eBPF binary file;
[0026] The loading function is called to load the eBPF binary into the kernel and verify it to compile the corresponding machine code;
[0027] The machine code is executed to provide an interface for calling an eBPF program.
[0028] Preferably, the requesting application synchronously triggers an eBPF program in the kernel, including:
[0029] In the kernel, the eBPF program trigger signal is received, and a callback function is issued to execute the eBPF program.
[0030] Preferably, at the user layer, the request source application requests to call a target application and receives a response from the target application. The log storage center consumes the message queue and processes and stores it in the log storage server so as to perform link tracing of the application call in the log storage server according to the link identifier.
[0031] Preferably, the step of performing link tracing on application calls in the log storage server based on the link identifier includes:
[0032] On the log storage server, the link information is retrieved based on the link identifier for analysis or link diagnosis.
[0033] This invention also provides an application call tracing system, including an application client and a log storage server:
[0034] The application includes:
[0035] The preprocessing module is used to pre-provide a calling interface for an eBPF program, which is used to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application.
[0036] The application operation module is used to request a source application to invoke a target application and receive a response from the target application.
[0037] The link information generation module is used to synchronously trigger an eBPF program in the kernel of the requesting source application, generate a unique link identifier, associate the basic information and response information of the requesting source application and the target application, and write them into the message queue to generate link information.
[0038] The log storage server includes a log storage center that consumes the message queue and processes and stores it in the log storage server to perform link tracing of application calls in the log storage server based on the link identifier.
[0039] The present invention also provides a distributed system, characterized in that: the above-mentioned application call tracing device is applied.
[0040] Compared with existing technologies, the above technical solution has the following advantages:
[0041] The tracing method provided in this application uses an eBPF program call interface provided in the kernel to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application. When an application request or response is executed, the eBPF program is triggered to generate unique link information. The log storage server can obtain the link information for storage or analysis, thereby realizing tracing and system problem diagnosis. Based on the eBPF program, the method realizes full-link tracing of a distributed application system, achieving non-intrusive tracing process to the application, reducing resource consumption, and improving application processing efficiency. Attached Figure Description
[0042] Figure 1 This is a flowchart of an embodiment of the application call chain tracing method described in this invention;
[0043] Figure 2 This is a schematic diagram illustrating the implementation process of an embodiment of the application call chain tracing method described in this invention;
[0044] Figure 3 This is a flowchart illustrating the generation of link information in an embodiment of the application call link tracing method described in this invention;
[0045] Figure 4 This is a schematic diagram of a module of an application call link tracing device according to the present invention.
[0046] Figure label:
[0047] 4-Application call link tracing device; 41-Application end; 42-Log storage server; 411-Preprocessing module; 412-Application operation module; 413-Link information generation module. Detailed Implementation
[0048] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.
[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0050] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0051] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0052] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0053] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0054] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.
[0055] Example 1: This example discloses an application call chain tracing method, see below. Figure 1 and Figure 2 This includes the following steps:
[0056] A pre-defined interface for calling an eBPF (Extended Berkeley Packet Filter) program is provided to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application.
[0057] In this implementation, the eBPF program runs in a non-intrusive manner in various environments such as physical networks, virtualization, and containers. By collecting data in the kernel, it improves data processing efficiency and reduces the consumption of computing resources and bandwidth consumption when outputting indicator data.
[0058] Before running an eBPF program, it needs to be compiled for invocation. Therefore, the aforementioned pre-provided eBPF program invocation interface includes:
[0059] The command is executed to compile the source file of the eBPF program into an eBPF binary file; the loading function is called to load the eBPF binary file into the kernel and verify that the corresponding machine code has been compiled; the machine code is executed to provide a calling interface for an eBPF program.
[0060] Furthermore, after compiling the source files and generating the calling interface, it can be directly called after subsequent triggering. That is, the request source application synchronously triggers the eBPF program in the kernel, including: receiving the eBPF program trigger signal in the kernel and issuing a callback function to execute the eBPF program.
[0061] As an example, the compiler command is run to compile sys_log_trace.c. clang is used to compile the eBPF source file sys_log_trace.c into eBPF bytecode sys_log_trace.o. The function load_bpf_file loads an eBPF binary file into the kernel, verifies it, and compiles it into the corresponding machine code. After execution, sys_log_trace.c becomes effective. When an application calls it subsequently, the kernel will trigger the bound callback function to execute the sys_log_trace.c program.
[0062] It should also be noted that because the Linux kernel uses the GPL license, it can only load programs licensed under the GPL. If a program is licensed under another license, the kernel will refuse to load it. Therefore, before the requesting application synchronously triggers the eBPF program in the kernel, the process includes: pre-setting the GPL license in the kernel and loading the GPL-licensed program before triggering the eBPF program. By loading the license of the specified program, the implementation of the eBPF program call is ensured.
[0063] S100: A source application requests to invoke a target application and receives a response from the target application;
[0064] Specifically, at the user level (user mode), the request source application requests to call a target application and receives a response from the target application. In step S300 below, the log storage center consumes (e.g., obtains, retrieves, etc.) the message queue and processes and stores it in the log storage server so as to perform link tracing of the application call in the log storage server according to the link identifier.
[0065] In this embodiment, it can be applied to a distributed system, which may include several applications. The calls between the applications and data processing (such as message queues and data storage) are implemented in the system application layer, while the eBPF program used to implement tracing requests is implemented in the kernel of the application side. It does not intrude on the application services, reduces program dependencies and runtime burden, and can significantly reduce resource consumption.
[0066] S200: The requesting source application synchronously triggers an eBPF program in the kernel to generate a unique link identifier, associates the basic information and response information of the requesting source application and the target application, and writes it into the message queue to generate link information.
[0067] In the above steps, a unique identifier is generated as trace_id to mark each response or request process. Different links can be traced based on the unique link identifier. In this embodiment, the requesting source application can request multiple target applications or contain multiple requests. A unique link identifier is marked for each request and its corresponding response.
[0068] For example, see Figure 2 Application A's method B calls application C's method D and receives a response. During the call, the kernel where application A resides triggers the eBPF program in sys_log_trace.c, executes its internal program, retrieves relevant information, and writes it to the message queue.
[0069] Specifically, the above process generates a unique link identifier, associates it with the basic information and response information of the request source application and the target application, and writes it to a message queue to generate link information, such as... Figure 3 ,include:
[0070] S210: Obtain basic information about the requesting source application and the target application, and add a unique link identifier;
[0071] Specifically, the basic information of both the requesting source application and the target application can include at least one of the following: application name; method; port; IP address; current system time, etc. In some embodiments, only one type of information may be obtained, while in other embodiments, multiple types of information may be obtained. This is not limited here. For example, obtaining basic information such as the name (A), method (B), port, IP address, and current system time of the requesting source application; and obtaining basic information such as the name (C), method (D), port, IP address, and current system time of the requesting target application.
[0072] S220: Collect network status codes, network response messages, target program response status codes and response messages, and current system time to obtain response information;
[0073] In the above steps, the response information may include not only the result information of the target response to the above request: network status code, network response message, target program response status code and response message (in the target application), and current system time, but also the following calculation of response time, so as to be written to the message queue later.
[0074] S230: Calculate the response time based on the collection time of the response information and the collection time of the basic information of the requesting source application;
[0075] In the above steps, the time of collecting the response information is the time of the target application's response, the time of collecting the basic information of the requesting source application is the time of sending the request, and the time between sending the request and receiving the target application's response is the aforementioned response time.
[0076] S240: After the response time, the basic information and response information are sequentially written into the message queue to generate link information corresponding to the unique link identifier.
[0077] That is, in the above steps, unique link information can be generated for each request.
[0078] As an example, the basic structure for writing to a message queue can be found in the table below:
[0079]
[0080]
[0081] S300: The log storage center consumes the message queue and processes and stores it in the log storage server to perform link tracing of application calls in the log storage server based on the link identifier.
[0082] In this implementation, the log storage center consumes data from the message queue, processes it, and then stores it in the log storage service. The system's operational status is tracked through log analysis and search services, and the entire request process can be traced based on the traceId identifier. This processing is implemented at the application layer. The eBPF program collects the aforementioned data in the kernel and sends it to the application layer for analysis, diagnosis, or storage, further reducing resource consumption and improving system application efficiency.
[0083] Specifically, the step of performing link tracing on application calls in the log storage server based on the link identifier includes: retrieving the link information based on the link identifier on the log storage server, and performing analysis or link diagnosis. Besides being used for request link tracing, it can also be used for system problem diagnosis, judging the system's operating status through network status or response status codes, thereby monitoring the operating status of each requesting application or target application in the system to improve system operating efficiency.
[0084] The link tracing method provided in this implementation sets up eBPF program calls under the kernel, running in a zero-intrusion manner in various environments such as physical networks, virtualization, and containers. It collects data in a near-kernel manner, achieving 10 times the performance of the existing component tcpdump. The computational resource consumption can be as low as 1% of the total resources of the running environment, and the bandwidth consumption when outputting indicator data can be as low as 1% of the business traffic. It can collect data from various services including middleware, storage, network, and self-developed applications, achieving one-click deployment, completely zero intrusion, and global applicability.
[0085] Example 2: This example also provides an application call tracing device 4, including an application client 41 and a log storage server 42:
[0086] The application terminal 41 includes:
[0087] The preprocessing module 411 is used to pre-provide a calling interface for an eBPF program, which is used to trigger the generation of a unique link identifier and associate the basic information and response information of the request source application and the target application.
[0088] In the preprocessing module, commands are run to compile the source files of the eBPF program, converting them into eBPF binary files. A loading function is then called to load the eBPF binary file into the kernel, and the corresponding machine code is verified and compiled. The machine code is executed to provide an interface for calling an eBPF program. After compiling the source files and generating the interface, the eBPF program can be directly called upon subsequent triggering. Before triggering the eBPF program, the GPL license is preloaded to ensure program execution.
[0089] Application operation module 412 is used for a source application to request and invoke a target application, and to receive a response from the target application;
[0090] Specifically, at the user layer, a request from a source application calls a target application. Each application can contain one or more source applications to simultaneously or in stages call one or more target applications. This can also be applied to clustered servers or distributed systems, which may include multiple application clients. Calls between these application clients and data processing (such as message queues and data storage) are implemented at the application layer. For example... Figure 2 Application A (the requesting source application) and Application C (the requesting source application) can be located on the same application platform or on different application platforms.
[0091] Link information generation module 413 is used to synchronously trigger an eBPF program in the kernel of the requesting source application, generate a unique link identifier, associate the basic information and response information of the requesting source application and the target application, and write the link information into a message queue.
[0092] During the call, the link information triggers the eBPF program. The generation module executes its internal program, obtains the relevant information, and writes it to the message queue to generate unique link information for each request and response.
[0093] Log storage server 42 includes a log storage center that consumes the message queue and processes and stores it in the log storage server to perform link tracing of application calls in the log storage server based on the link identifier.
[0094] As can be seen, the log storage server obtains data from the eBPF program and performs operations such as link analysis, system problem diagnosis, and judging the system operation status through network status or response status codes. It is deployed outside the application, that is, in the distributed system application layer, at the same level as the requesting application and the target application.
[0095] In this embodiment, the source code of the eBPF program is compiled by the kernel execution command through the preprocessing module. An eBPF program calling interface is provided in advance to trigger the generation of a unique link identifier and associate it with the basic information and response information of the requesting source application and the target application. When the application operation module executes the application request or response, the eBPF program is triggered to generate a unique link identifier and associate it with the basic information and response information of the requesting source application and the target application. The unique link information is written to the message queue. The log storage server can obtain the link information, store or analyze it, and realize link tracing, system problem diagnosis, and system operation status judgment through network status or response status code. Based on the eBPF program, the full link tracing of the distributed application system is realized, reducing component insertion, reducing the occupation of system resources, realizing non-intrusive tracing process to the application, and improving application processing efficiency.
[0096] Example 3: This example provides a distributed system that uses the application call tracing device described in Example 2. The distributed system includes several application terminals and other servers or control modules for system operation. Each application terminal can be a request source application or a target application, or it can contain multiple request source applications or target applications. Each request source application can send several requests to different target applications. Each request can obtain link information with a unique link identifier through the method in Example 1, thereby performing link analysis, system problem diagnosis and other functions in the log storage server.
[0097] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. An application call tracing method, characterized in that, include: A pre-defined eBPF application call interface is provided to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application. A source application requests to invoke a target application and receives a response from the target application; The requesting application synchronously triggers an eBPF program in the kernel to generate a unique link identifier, associates the basic information and response information of the requesting application and the target application, and writes it into the message queue to generate link information. The log storage center consumes the message queue and processes and stores it in the log storage server, so as to perform link tracing of application calls in the log storage server according to the link identifier; The process of generating a unique link identifier, associating it with the basic information and response information of the requesting source application and the target application, and writing it into a message queue to generate link information includes: Obtain basic information about the request source application and the target application, and add a unique link identifier; Collect network status codes, network response messages, target program response status codes and response messages, and the current system time to obtain response information; The response time is calculated based on the collection time of the response information and the collection time of the basic information of the requesting source application. After the response time, the basic information and response information are sequentially written into the message queue to generate link information corresponding to the unique link identifier; The basic information of both the requesting source application and the target application includes at least one of the following: Application name; method; port; IP; Current system time.
2. The link tracing method according to claim 1, characterized in that, Before the requesting source application synchronously triggers the eBPF program in the kernel, the following is also included: The GPL license is pre-configured in the kernel, and the GPL-licensed program is loaded before the eBPF program is triggered.
3. The link tracing method according to claim 1, characterized in that, The pre-provided eBPF program calling interface includes: Run the command to compile the source file of the eBPF program, and compile the source file of the eBPF program into an eBPF binary file; The loading function is called to load the eBPF binary into the kernel and verify it to compile the corresponding machine code; The machine code is executed to provide an interface for calling an eBPF program.
4. The link tracing method according to claim 1, characterized in that, The requesting application synchronously triggers an eBPF program in the kernel, including: In the kernel, the eBPF program trigger signal is received, and a callback function is issued to execute the eBPF program.
5. The link tracing method according to claim 1, characterized in that: The user layer executes the request from the source application to call a target application and receives the response from the target application. The log storage center consumes the message queue, processes and stores it in the log storage server, so as to perform link tracing of the application call in the log storage server according to the link identifier.
6. The link tracing method according to claim 5, characterized in that, The step of tracing application calls in the log storage server based on the link identifier includes: On the log storage server, the link information is retrieved based on the link identifier for analysis or link diagnosis.
7. An application call link tracing device, characterized in that, This includes the application itself and the log storage server: The application includes: The preprocessing module is used to pre-provide a calling interface for an eBPF program, which is used to trigger the generation of a unique link identifier and associate basic information and response information of the request source application and the target application. The application operation module is used to request a source application to invoke a target application and receive a response from the target application. The link information generation module is used to synchronously trigger an eBPF program in the kernel of the requesting source application, generate a unique link identifier, associate the basic information and response information of the requesting source application and the target application, and write them into the message queue to generate link information. The log storage server is used by the log storage center to consume the message queue and process and store it in the log storage server, so as to perform link tracing of application calls in the log storage server according to the link identifier; The process of generating a unique link identifier, associating it with the basic information and response information of the requesting source application and the target application, and writing it into a message queue to generate link information includes: Obtain basic information about the request source application and the target application, and add a unique link identifier; Collect network status codes, network response messages, target program response status codes and response messages, and the current system time to obtain response information; The response time is calculated based on the collection time of the response information and the collection time of the basic information of the requesting source application. After the response time, the basic information and response information are sequentially written into the message queue to generate link information corresponding to the unique link identifier; The basic information of both the requesting source application and the target application includes at least one of the following: Application name; method; port; IP; Current system time.
8. A distributed system, characterized in that: The application call link tracing device described in claim 7 above is applied.