Application service data flow analysis method and system based on eBPF

Abstract

The invention discloses an eBPF-based application service data flow analysis method and system, and relates to the technical field of application service data flow, and the method comprises the following steps: initializing a probe program; the probe program obtains calling relation data of the whole application service from the system through the eBPF technology and sends the calling relation data to the server side; the probe program obtains data related to database operation from the protocol aspect through the eBPF technology and sends the data to the server side; the server side obtains calling relation data and data related to database operation; and the server side stores the received data in a persistent memory and processes the data through a data analysis engine. By adopting the eBPF technology, the accurate service calling relation and database operation details can be directly obtained from an operating system kernel under the condition that application program codes are not modified, the influence on application performance is reduced, and higher accuracy and instantaneity are provided.

Description

Technical Field

[0001] The present invention relates to the technical field of application business data flow, and more specifically to an application business data flow analysis method and system based on eBPF. Background Art

[0002] With the deepening of enterprise digital transformation and the continuous emergence and application of emerging technologies, the integration of systems used within enterprises is constantly improving. The systems used within enterprises are often huge in scale, complex in structure, and distributed in architecture. The current system observability mainly achieves operational observation and root cause analysis from aspects such as logging, performance indicators, and distributed tracing, but cannot achieve good application business data flow analysis.

[0003] Technology Status

[0004] Traditional monitoring mainly collects key log and indicator information at the system layer and application layer. Some applications also need to adjust the code, configuration and account information of the application service to obtain monitoring data. The method of obtaining data is costly and inefficient, and will reduce the operating efficiency of the entire business application service, which has a great impact on the production environment.

[0005] The current defects in the application of business data flow analysis methods are:

[0006] Lack of dynamic tracking and flexibility: Traditional systems can only rely on static analysis and post-mortem log analysis, which limits flexibility and real-time performance.

[0007] High performance overhead: Traditional system monitoring and data collection methods, such as kernel modules or external packet capture tools, often have a significant impact on system performance.

[0008] Difficulty in capturing fine-grained data: The system may only be able to obtain relatively coarse-grained monitoring data, which limits the ability to deeply analyze the flow of business data.

[0009] Limited real-time analysis: The system may rely on log analysis or batch data processing methods with high latency, which will affect the effectiveness of real-time analysis.

[0010] Difficult visualization and debugging: It may be more difficult for developers and operation and maintenance personnel to understand and debug business data flow problems in the system.

[0011] Therefore, it is necessary to propose an application business data flow analysis method and system based on eBPF to solve the above problems. Summary of the invention

[0012] The purpose of the present invention is to solve the problems raised in the background technology.

[0013] The present invention specifically adopts the following technical solutions to achieve the above object:

[0014] An eBPF-based method for analyzing the data flow of application services, comprising the following steps:

[0015] S1. Initialize the probe program;

[0016] S2. The probe program obtains the call relationship data of the entire application service from the system through eBPF technology and sends it to the server;

[0017] S3. The probe program obtains the data related to database operations from the protocol aspect through eBPF technology and sends it to the server side;

[0018] S4. The server side obtains the call relationship data and the data related to database operations;

[0019] S5. The server side stores the received call relationship data and the data related to database operations in persistent storage and processes them through a data analysis engine to generate a visualization report or an alarm.

[0020] Further, the specific steps of the call relationship data in S2 are as follows: when an application program initiates a function call or a service request, the eBPF probe captures these events and extracts the call link information.

[0021] Further, the specific steps of obtaining the data related to database operations in S3 are as follows: for operations related to the database, the eBPF probe intercepts the relevant protocol packets and parses the SQL statements and other metadata therein.

[0022] Further, the call relationship data and the data related to database operations obtained in S2 and S3 are formatted and sent to the server side through a secure channel.

[0023] Further, the probe program initialization step in S1: load a predefined eBPF program into the kernel space of the target system at startup to prepare for listening for specified events.

[0024] An eBPF-based application service data flow analysis system, including the above eBPF-based application service data flow analysis method, further includes:

[0025] A probe program, deployed on each application node, using eBPF technology to penetrate into the operating system kernel layer to capture the call relationships and database operations at the application layer;

[0026] A network topology, including client devices, application servers, database servers, and a central server for collecting and analyzing data;

[0027] The server side is responsible for receiving data from the probe program, storing, processing, and displaying it.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. By adopting the eBPF technology, the present invention can directly obtain accurate business call relationships and database operation details from the operating system kernel without modifying the application program code, reducing the impact on application performance and providing higher accuracy and real-time performance.

[0030] 2. The present invention utilizes the advanced Linux kernel feature of eBPF to achieve efficient monitoring of the application business data flow. Compared with traditional monitoring means, this method has lower invasiveness, higher accuracy, and better performance, and is particularly suitable for performance debugging and fault troubleshooting in modern distributed system environments.

[0031] 3. The present invention uses the eBPF technology to directly obtain call relationship data and database operation data from the system bottom layer and send this information to the server side for centralized processing and analysis. This method not only improves the accuracy and real-time performance of data capture but also avoids the performance overhead brought by traditional monitoring means.

[0032] 4. The present invention provides a non-invasive solution that can capture detailed business logic and database interaction information at the operating system kernel level without modifying the application program code. This greatly enhances the observability of the system and facilitates developers and operation and maintenance teams to conduct fault troubleshooting and performance optimization.

[0033] 5. Through accurate data collection and analysis, the present invention can help enterprises better understand their resource usage, reasonably plan resource allocation, avoid resource waste, and improve the utilization rate of the overall IT infrastructure. Description of the Drawings

[0034] Figure 1 It is a flowchart of a method for analyzing the application business data flow in the present invention.

[0035] Figure 2 It is a module diagram of the application business data flow analysis system in the present invention.

[0036] Figure 3 It is another flowchart of the method for analyzing the application business data flow in the present invention.

[0037] Figure 4 It is a signaling interaction diagram between the probe program and the server side in the present invention. Detailed Embodiments

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] Please refer to Figures 1-4 , an eBPF-based application service data flow analysis system, including:

[0040] Probe programs, deployed on each application node, using eBPF technology to deeply penetrate into the operating system kernel layer to capture the call relationships and database operations at the application layer;

[0041] Network topology, including client devices, application servers, database servers, and a central server for collecting and analyzing data;

[0042] Server side, responsible for receiving data from the probe programs, storing, processing, and displaying it.

[0043] An eBPF-based application service data flow analysis method includes the following steps:

[0044] S1. Initialize the probe program. When starting, load the predefined eBPF program into the kernel space of the target system, prepare to listen for specified events, events related to protocols, such as ATTP, message queues, etc.;

[0045] S2. The probe program obtains the call relationship data of the entire application service from the system through eBPF technology. When the application program initiates a function call or service request, the eBPF probe captures these events, extracts the call link information, and sends it to the server side;

[0046] S3. The probe program obtains the data related to database operations from the protocol aspect through eBPF technology. For operations involving databases, the eBPF probe intercepts the relevant protocol packets, parses the SQL statements and other metadata in them. The metadata includes but is not limited to: timestamp: the time when the data packet is captured; source and target IP addresses: the IP address of the client initiating the request and the IP address of the server receiving the request; port numbers: the source port and target port of the communication; protocol type: such as TCP, UDP, etc.; database type: such as MySQL, PostgreSQL, etc.; client information: such as client version or identifier; transaction ID: the unique identifier of the database transaction; response status: such as the success or failure status of the SQL query; execution time: the execution duration of the SQL statement; data packet size: the size of the request or response data packet, and sends it to the server side;

[0047] After the call relationship data and the data related to database operations obtained in S2 and S3 are formatted, they are sent to the server side through a secure channel;

[0048] S4. The server side obtains the call relationship data and the data related to database operations;

[0049] S5. The server side stores the received call relationship data and the data related to database operations into persistent storage and processes them through a data analysis engine to generate a visualization report or an alert.

[0050] Among them, eBPF (Extended Berkeley Packet Filter): A Linux kernel technology that allows running sandboxed program fragments to perform various tasks, such as traffic filtering, performance analysis, etc., without changing the kernel source code or loading modules.

[0051] Probe program: Refers to a lightweight monitoring program written using eBPF, embedded in the operating system kernel, used to capture specific events and generate corresponding outputs.

[0052] eBPF allows dynamic modification and injection of tracepoints at runtime, enabling the system to collect detailed business data flow information without affecting performance.

[0053] eBPF can efficiently run user-defined code at the kernel level, reducing performance overhead and making the tracking and analysis of business data streams more efficient.

[0054] eBPF can capture and analyze fine-grained events in multiple kernel subsystems such as the network stack, file system, and process scheduling.

[0055] eBPF can be integrated with various monitoring and tracing tools and has strong scalability.

[0056] eBPF can achieve near-real-time data analysis and anomaly detection, which is crucial for quickly responding to business changes.

[0057] The dynamic data collection and analysis capabilities provided by eBPF help to achieve the visualization and debugging of complex business logic.

[0058] Scenario example

[0059] Suppose in an e-commerce platform, the process of a user placing an order to purchase a product triggers a series of calls between microservices, including inventory checking, price calculation, payment gateway communication, etc. At the same time, these operations also involve multiple database reads and writes. Through the method of the present invention, we can track all call links in real time during the processing of each order and accurately record the corresponding SQL queries for each step, thereby helping the development team quickly locate the root cause of problems and optimize system performance.

[0060] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. The patent protection scope of the present invention is subject to the claims. All equivalent structural changes made by using the content of the specification and drawings of the present invention should, by the same token, be included in the protection scope of the present invention.

Claims

1. The eBPF-based application service data flow analysis method is characterized by: The following steps are involved: S1, probe program initialization; S2. The probe program obtains the call relationship data of the entire application business from the system through eBPF technology and sends it to the server; S3. The probe program obtains data related to database operations from the protocol through eBPF technology and sends it to the server. S4, the server obtains the call relationship data and the data related to the database operation; S5. The server stores the received call relationship data and the data related to the database operation in the persistent storage, and processes them through the data analysis engine to generate a visual report or alarm.

2. The eBPF-based application service data flow analysis method according to claim 1, characterized in that: The specific steps of the call relationship data in S2 are: when the application initiates a function call or service request, the eBPF probe captures these events and extracts the call link information.

3. The eBPF-based application service data flow analysis method according to claim 1, characterized in that: The specific steps of obtaining data related to database operations in S3 are as follows: for operations related to the database, the eBPF probe will intercept the relevant protocol packets and parse the SQL statements and other metadata therein.

4. The eBPF-based application service data flow analysis method according to claim 1, characterized in that: The call relationship data and the data related to the database operation obtained in S2 and S3 are formatted and sent to the server through a secure channel.

5. The eBPF-based application service data flow analysis method according to claim 1, characterized in that: The probe program initialization step in S1 is as follows: upon startup, a predefined eBPF program is loaded into the kernel space of the target system, ready to monitor specified events.

6. An application service data flow analysis system based on eBPF, comprising the application service data flow analysis method based on eBPF according to any one of claims 1 to 5, characterized in that: Also includes: The probe program is deployed on each application node and uses the eBPF technology to penetrate into the kernel layer of the operating system to capture the call relationship and database operations of the application layer. Network topology, including client devices, application servers, database servers, and central servers that collect and analyze data; The server side is responsible for receiving data from the probe program and storing, processing and displaying it.

Images