Gateway fault positioning system and method based on eBPF, and gateway

Abstract

The invention relates to the technical field of communication, discloses an eBPF-based gateway fault positioning system and method, and a gateway, and aims to solve the problems of low accuracy and efficiency and influence on normal service processing in the existing method, and the scheme mainly comprises the following steps: an instruction module provides a man-machine interaction inlet; after the alarm management module judges that a gateway fault occurs, the alarm management module matches and associates to a preset eBPF program capable of collecting required key kernel data, and sends a corresponding loading task instruction to an eBPF loading module according to the associated eBPF program; the eBPF loading module loads a corresponding eBPF program to a kernel according to the loading task instruction, and when the eBPF program runs in the kernel, corresponding key kernel data is captured and sent to the data management module; and the data management module organizes the key kernel data and outputs the organized key kernel data in real time. According to the method, the accuracy and the efficiency of gateway fault positioning are improved, the influence on normal service processing is avoided, and the method is suitable for various gateways.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an eBPF-based gateway fault location system and method, and a gateway. Background Art

[0002] Gateways are critical network infrastructure, connecting different networks, protocols, or systems. Their core role is to bridge communications between different network domains, ensuring data is transmitted correctly across these differences. When network traffic passes through a gateway, a gateway failure can lead to connection failures, high latency, and incorrect responses. By locating and analyzing the root cause of a gateway failure, we can quickly resolve service availability issues and performance bottlenecks caused by gateway failures.

[0003] The existing gateway fault location method usually exports and saves business logs after the gateway fails, and performs preliminary fault location based on the business logs. If the log analysis yields no results, it is converted to a complex fault. When encountering a complex fault, try to reproduce the fault phenomenon. Technicians can modify the program and use various tools to reproduce the fault phenomenon in a laboratory environment and perform analysis and judgment. For some complex faults that can only be reproduced in a field environment, more comprehensive log analysis can be obtained by modifying the log level.

[0004] The above solution has at least the following defects: First, although the gateway uses business logs to record the running status of the gateway, the business logs have data blind spots and provide limited effective information. It is impossible to obtain key kernel behavior data through business logs, and it is difficult for technicians to accurately locate gateway faults occurring in the kernel directly based on business logs; second, reproducing the fault phenomenon in a laboratory environment requires interrupting the gateway service, which cannot guarantee the continuity of the gateway service and affects normal business processing. In addition, technicians need to repeatedly modify the code, replace the program for debugging on site, and observe for a long time. The time cost of fault location is high and the efficiency is poor; third, modifying the log level to obtain more comprehensive log analysis will affect normal business performance and reduce business processing capabilities. Summary of the Invention

[0005] The present invention aims to solve the problems of low accuracy and efficiency of existing gateway fault location methods and the impact on normal business processing, and proposes a gateway fault location system and method based on eBPF, and a gateway.

[0006] The technical solution adopted by the present invention to solve the above technical failure is: In a first aspect, the present invention provides a gateway fault location system based on eBPF, the system comprising: Instruction module, used to provide a human-computer interaction entrance; The alarm management module is used to match and associate the preset eBPF program that can collect the required key kernel data after determining that a gateway failure has occurred, and send the corresponding loading task instructions to the eBPF loading module according to the associated eBPF program; An eBPF loading module is used to load the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module; The data management module is used to organize the key kernel data and output the organized key kernel data in real time.

[0007] Furthermore, determining that a gateway failure occurs includes: Preset multiple warning rules in the alarm management module through the instruction module; The alarm management module automatically determines that a gateway failure occurs when the operation index of the gateway meets a certain early warning rule.

[0008] Furthermore, matching and associating with a preset eBPF program that can collect the required key kernel data includes: Pre-establishing a mapping relationship between an early warning rule and an eBPF program identifier in the alarm management module, wherein the eBPF program identifier is used to indicate an eBPF program capable of collecting key kernel data required for the corresponding early warning rule; When the operation index of the gateway meets the warning rule, the alarm management module matches the eBPF program identifier corresponding to the warning rule based on the mapping relationship and generates a loading task instruction including the eBPF program identifier.

[0009] Furthermore, determining that a gateway failure occurs includes: After receiving the early warning instruction sent by the instruction module, the alarm management module automatically determines that a gateway failure has occurred.

[0010] Furthermore, matching and associating with a preset eBPF program that can collect the required key kernel data includes: Pre-establishing a mapping relationship between an early warning instruction and an eBPF program identifier in the alarm management module, wherein the eBPF program identifier is used to indicate an eBPF program capable of collecting key kernel data required for the corresponding early warning instruction; After receiving the early warning instruction, the alarm management module generates a loading task instruction including the eBPF program identifier.

[0011] Furthermore, loading the corresponding eBPF program into the kernel according to the loading task instruction includes: Pre-store multiple compiled eBPF programs in the eBPF object repository; After receiving the loading task instruction, the eBPF loading module retrieves the corresponding eBPF program from the eBPF object repository, loads the retrieved eBPF program into the kernel, and establishes a data channel from the eBPF program to the data management module.

[0012] Furthermore, the data management module is specifically used to: Select the corresponding parsing logic based on the source of key kernel data, parse the key kernel data and extract structured information, filter and aggregate the structured information, and generate organized key kernel data; The organized key kernel data is displayed in real time and saved to a log file.

[0013] Furthermore, the gateway failure includes: excessive CPU usage, excessive memory usage and communication abnormality; the key kernel data includes: packet queue data, buffer data and connection release speed data.

[0014] In a second aspect, the present invention provides a gateway fault location method based on eBPF, which is applied to the gateway fault location system based on eBPF as described in the first aspect, and the method includes: After determining that a gateway failure has occurred, the alarm management module matches and associates it with a preset eBPF program that can collect the required key kernel data, and sends the corresponding loading task instructions to the eBPF loading module based on the associated eBPF program; The eBPF loading module loads the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module; The data management module organizes the key kernel data and outputs the organized key kernel data in real time.

[0015] In a third aspect, the present invention provides a gateway, comprising the eBPF-based gateway fault location system as described in the first aspect.

[0016] The beneficial effects of the present invention are as follows: the gateway fault location system and method based on eBPF, and the gateway provided by the present invention automatically determine whether a gateway fault occurs. After determining that a gateway fault occurs, the associated eBPF program is automatically dynamically loaded in the kernel. The eBPF program takes effect immediately after a security check by the kernel verifier, without interrupting the gateway service, thereby avoiding the impact on normal business processing; the eBPF program is mounted on the kernel critical path, and its hot loading mechanism bypasses the kernel compilation dependency, and can directly capture kernel key data that cannot be obtained through business logs, thereby achieving accurate positioning and collection of kernel key data and improving fault location efficiency; technicians can quickly and accurately locate gateway faults based on key kernel data, especially gateway faults occurring in the kernel, thereby avoiding the data blind spot problem in the business log, improving the efficiency and accuracy of gateway fault location, and also avoiding the performance impact and storage redundancy caused by modifying the log level. In addition, the eBPF program preloads filtering rules through the Maps structure and only returns valid data, thereby avoiding sampling distortion and further avoiding storage redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the structure of a gateway fault location system based on eBPF provided in an embodiment; Figure 2 A schematic diagram of a flow chart of a gateway fault location method based on eBPF provided in an embodiment; Figure 3 A timing diagram of a gateway fault location method based on eBPF provided in an embodiment; Figure 4 A timing diagram of another eBPF-based gateway fault location method provided in an embodiment. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is applicable to application scenarios where gateway fault location needs to be performed, for example, fault location of gateways such as API gateways, cloud gateways, and border gateways.

[0019] Current gateway fault location methods typically analyze simple faults based on service logs. When encountering complex faults, reproducing the fault symptoms and modifying the log level can yield more comprehensive service log analysis. However, due to the limited data captured by service logs, it's difficult to accurately locate gateway faults occurring in the kernel directly based on service logs. Furthermore, reproducing the fault symptoms is time-consuming and inefficient. Furthermore, modifying the log level can impact normal service performance.

[0020] Based on this, the technical solution of the present invention is proposed. In the present invention, the alarm management module automatically determines whether the gateway has failed. After determining that the gateway has failed, it automatically generates a loading task instruction for the associated eBPF program; the eBPF loading module automatically and dynamically loads the associated eBPF program in the kernel according to the loading task instruction; because the eBPF program is mounted on the kernel critical path, its hot loading mechanism bypasses the kernel compilation dependency and can directly capture kernel critical data that cannot be obtained through business logs. In addition, the eBPF program takes effect immediately after the kernel verifier security check, without interrupting the gateway service. The technician can quickly and accurately locate the gateway fault based on the key kernel data output by the data management module. In addition, rule association ensures that the collected data directly corresponds to the fault, achieving accurate data positioning.

[0021] The technical solution of this embodiment will be clearly and completely described below in conjunction with the drawings in this embodiment. Obviously, the described embodiment is only a part of the embodiments of the present invention, rather than all the embodiments.

[0022] Figure 1 A gateway fault location system based on eBPF is shown, see Figure 1 ,The system includes: an instruction module, an alarm management module, an eBPF loading module and a data management module.

[0023] Among them, the instruction module is used to provide a human-computer interaction entrance. The alarm management module is used to match and associate with a preset eBPF program that can collect the required key kernel data after determining that a gateway failure has occurred, and send a corresponding loading task instruction to the eBPF loading module according to the associated eBPF program. The eBPF loading module is used to load the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module. The data management module is used to organize the key kernel data and output the organized key kernel data in real time.

[0024] Based on the above system, this embodiment proposes a gateway fault location method based on eBPF, please refer to Figure 2 , the method comprises the following steps: Step 1: After determining that a gateway failure has occurred, the alarm management module matches and associates it with a preset eBPF program that can collect the required key kernel data, and sends corresponding loading task instructions to the eBPF loading module according to the associated eBPF program.

[0025] eBPF (Extended Berkeley Packet Filter) is a sandboxed virtual machine running in the Linux kernel. It allows developers to dynamically inject security-verified bytecode programs into critical kernel paths (such as system calls, network protocol stacks, and function entry / exit points) without modifying the kernel source code or restarting the system. This enables programmable observation and control of kernel behavior, thus eliminating blind spots in service logs.

[0026] In this embodiment, there are two methods for determining whether a gateway failure occurs.

[0027] See also Figure 3 The first method for determining whether a gateway failure occurs includes: presetting multiple warning rules in the alarm management module through the instruction module; the alarm management module automatically determines that a gateway failure occurs when the operation indicators of the gateway meet a certain warning rule.

[0028] In actual use, technicians use the command module to preset multiple warning rules in the alarm management module's rule base. The alarm management module receives gateway operating indicators collected by an external monitoring system (or a built-in indicator collector) and determines whether the gateway operating indicators meet a specific warning rule. If so, it automatically determines that a gateway failure has occurred. For example, if the CPU usage is 100% for a long period of time, or if memory usage increases and is not released, it will automatically determine that a gateway failure has occurred.

[0029] Based on the first method for determining whether a gateway failure has occurred, the system matches and associates the system with a preset eBPF program that can collect the required key kernel data, including: A mapping relationship between warning rules and eBPF program identifiers is pre-constructed in the alarm management module. The eBPF program identifier is used to indicate the eBPF program that can collect key kernel data required by the corresponding warning rules. When the operating indicators of the gateway meet the warning rules, the alarm management module matches the eBPF program identifier corresponding to the warning rule based on the mapping relationship and generates a loading task instruction containing the eBPF program identifier.

[0030] In actual applications, technicians use the instruction module to preset the mapping relationship between warning rules and eBPF program identifiers in the alarm management module. After the gateway's operating indicators meet a certain warning rule and the alarm management module determines that a gateway failure has occurred, it automatically generates a loading task instruction containing the eBPF program identifier corresponding to the warning rule and sends it to the eBPF loading module.

[0031] See also Figure 4 The second method for determining whether a gateway failure occurs includes: after the alarm management module receives the early warning instruction sent by the instruction module, it automatically determines that a gateway failure occurs.

[0032] In actual applications, technicians proactively discover gateway failures, such as network data communication failures. Without interrupting gateway services, technicians can proactively input early warning instructions through the instruction module. After receiving the early warning instructions, the alarm management module automatically determines that a gateway failure has occurred.

[0033] Based on the second method for determining whether a gateway failure has occurred, the system matches and associates the pre-set eBPF program that can collect the required key kernel data, including: A mapping relationship between the warning instruction and the eBPF program identifier is pre-constructed in the alarm management module, and the eBPF program identifier is used to indicate the eBPF program that can collect the key kernel data required for the corresponding warning instruction; after receiving the warning instruction, the alarm management module generates a loading task instruction containing the eBPF program identifier.

[0034] In actual applications, technicians use the instruction module to preset the mapping relationship between the warning instruction and the eBPF program identifier in the alarm management module. After receiving the warning instruction and determining that a gateway failure has occurred, the alarm management module automatically generates a loading task instruction containing the eBPF program identifier corresponding to the warning instruction and sends it to the eBPF loading module.

[0035] In this embodiment, the gateway failure includes but is not limited to: excessive CPU usage, excessive memory usage, and communication abnormalities; the key kernel data includes but is not limited to: packet queue data, buffer data, and connection release speed data.

[0036] Step 2: The eBPF loading module loads the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module.

[0037] In this embodiment, loading the corresponding eBPF program into the kernel according to the loading task instruction includes: A plurality of compiled eBPF programs are pre-stored in the eBPF object repository; after receiving the loading task instruction, the eBPF loading module retrieves the corresponding eBPF program from the eBPF object repository, loads the retrieved eBPF program into the kernel, and establishes a data channel from the eBPF program to the data management module.

[0038] In practice, technicians use the instruction module to pre-store multiple compiled eBPF programs in the eBPF object repository. Each eBPF program includes corresponding metadata (program type, Maps definition, and supported configuration parameters) for collecting critical kernel data. After the eBPF loader module receives the load task instruction from the alarm management module, it retrieves the corresponding eBPF program object from the repository based on the eBPF program identifier in the load task instruction and then loads it. First, it performs a bytecode security and compatibility check. The eBPF program is then loaded into the kernel, where it undergoes a rigorous check by the kernel verifier. Configuration parameters (such as filtering IP addresses and PIDs) are then written to the eBPF program's corresponding Maps, enabling the eBPF program to begin collecting data on demand. Based on the program type, the eBPF program is mounted to a designated kernel function, technician-mode function, tracepoint, or network hook. Finally, a data channel is established from the eBPF program's Maps to the user-mode data management module.

[0039] Because eBPF programs are mounted on the kernel's critical path, their hot reloading mechanism bypasses kernel compilation dependencies, enabling them to directly capture critical kernel data that's not available through service logs and send it to the data management module. Furthermore, eBPF programs take effect immediately after passing a kernel verifier security check, without interrupting gateway services.

[0040] Step 3: The data management module organizes the key kernel data and outputs the organized key kernel data in real time.

[0041] In this embodiment, the data management module is specifically used to: Select the corresponding parsing logic based on the source of the key kernel data, parse the key kernel data and extract structured information, filter and aggregate the structured information to generate organized key kernel data; display the organized key kernel data in real time and save it to a log file.

[0042] In actual applications, after receiving critical kernel data collected by the eBPF program, the data management module first parses the binary kernel event data according to the event type (defined by the associated eBPF program), extracting key fields (such as timestamp, process PID, function name, call stack, and packet length). It then performs preliminary data filtering or aggregation to generate organized critical kernel data. Finally, the organized critical kernel data is pushed to standard output, a socket, or a message queue for real-time display on a real-time monitoring terminal or stream processing system. The organized critical kernel data is also appended to a local log file for persistent data storage. Based on the critical kernel data output by the data management module, technicians can quickly and accurately locate gateway faults.

[0043] In summary, the eBPF-based gateway fault location system and method provided in this embodiment automatically determines whether a gateway fault occurs. After determining that a gateway fault occurs, the associated eBPF program is automatically dynamically loaded into the kernel. The eBPF program takes effect immediately after a security check by the kernel verifier, without interrupting the gateway service, thus avoiding the impact on normal business processing. The eBPF program is mounted on the kernel critical path, and its hot loading mechanism bypasses kernel compilation dependencies and can directly capture kernel key data that cannot be obtained through business logs, thereby achieving accurate positioning and collection of kernel key data and improving fault location efficiency. Technicians can quickly and accurately locate gateway faults based on key kernel data, especially gateway faults occurring in the kernel, avoiding the data blind spot problem in business logs, improving the efficiency and accuracy of gateway fault location, and also avoiding the performance impact and storage redundancy caused by modifying the log level. In addition, the eBPF program preloads filtering rules through the Maps structure and only returns valid data, thus avoiding sampling distortion and further avoiding storage redundancy.

[0044] Based on the above technical solution, this embodiment further proposes a gateway, which includes the eBPF-based gateway fault location system as described in the embodiment.

[0045] It can be understood that since the gateway described in this embodiment includes the eBPF-based gateway fault location system of the embodiment, for the gateway disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple. For relevant points, please refer to the partial description of the system, which will not be repeated here.

Claims

1. A gateway fault location system based on eBPF, characterized in that: The system comprises: Instruction module, used to provide a human-computer interaction entrance; The alarm management module is used to match and associate the preset eBPF program that can collect the required key kernel data after determining that a gateway failure has occurred, and send the corresponding loading task instructions to the eBPF loading module according to the associated eBPF program; An eBPF loading module is used to load the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module; The data management module is used to organize the key kernel data and output the organized key kernel data in real time.

2. The eBPF-based gateway fault location system according to claim 1, characterized in that: Determining whether a gateway failure has occurred includes: Preset multiple warning rules in the alarm management module through the instruction module; The alarm management module automatically determines that a gateway failure occurs when the operation index of the gateway meets a certain early warning rule.

3. The eBPF-based gateway fault location system according to claim 2, characterized in that: Match and associate with the preset eBPF program that can collect the required key kernel data, including: Pre-establishing a mapping relationship between an early warning rule and an eBPF program identifier in the alarm management module, wherein the eBPF program identifier is used to indicate an eBPF program capable of collecting key kernel data required for the corresponding early warning rule; When the operation index of the gateway meets the warning rule, the alarm management module matches the eBPF program identifier corresponding to the warning rule based on the mapping relationship and generates a loading task instruction including the eBPF program identifier.

4. The eBPF-based gateway fault location system according to claim 1, characterized in that: Determining whether a gateway failure has occurred includes: After receiving the early warning instruction sent by the instruction module, the alarm management module automatically determines that a gateway failure has occurred.

5. The eBPF-based gateway fault location system according to claim 4, characterized in that: Match and associate with the preset eBPF program that can collect the required key kernel data, including: Pre-establishing a mapping relationship between an early warning instruction and an eBPF program identifier in the alarm management module, wherein the eBPF program identifier is used to indicate an eBPF program capable of collecting key kernel data required for the corresponding early warning instruction; After receiving the early warning instruction, the alarm management module generates a loading task instruction including the eBPF program identifier.

6. The eBPF-based gateway fault location system according to claim 1, characterized in that: Loading the corresponding eBPF program into the kernel according to the loading task instruction includes: Pre-store multiple compiled eBPF programs in the eBPF object repository; After receiving the loading task instruction, the eBPF loading module retrieves the corresponding eBPF program from the eBPF object repository, loads the retrieved eBPF program into the kernel, and establishes a data channel from the eBPF program to the data management module.

7. The eBPF-based gateway fault location system according to claim 1, characterized in that: The data management module is specifically used for: Select the corresponding parsing logic based on the source of key kernel data, parse the key kernel data and extract structured information, filter and aggregate the structured information, and generate organized key kernel data; The organized key kernel data is displayed in real time and saved to a log file.

8. The eBPF-based gateway fault location system according to claim 1, characterized in that: The gateway failure includes: excessive CPU usage, excessive memory usage and communication abnormalities; the key kernel data includes: packet queue data, buffer data and connection release speed data.

9. A gateway fault location method based on eBPF, characterized in that: Applied to the eBPF-based gateway fault location system according to any one of claims 1 to 8, the method comprises: After determining that a gateway failure has occurred, the alarm management module matches and associates it with a preset eBPF program that can collect the required key kernel data, and sends the corresponding loading task instructions to the eBPF loading module based on the associated eBPF program; The eBPF loading module loads the corresponding eBPF program into the kernel according to the loading task instruction. When the eBPF program runs in the kernel, it captures the corresponding key kernel data and sends it to the data management module; The data management module organizes the key kernel data and outputs the organized key kernel data in real time.

10. A gateway, characterized in that: The gateway includes the eBPF-based gateway fault location system according to any one of claims 1 to 8.

Images