eBPF-based microservice request fault injection method and apparatus

Abstract

This invention discloses a microservice request fault injection method and apparatus based on eBPF. The method includes the following steps: monitoring the process of establishing a socket between the client and the server; capturing the socket and obtaining the port information of the socket; determining whether the socket is an object to be monitored and fault injected based on the port information of the socket; updating the socket as a value in SOCKHASH based on the determination result, and establishing an index with the corresponding port information as a key value; when the socket in SOCKHASH calls the sendfile() or sendmsg() function, performing fault injection at the socket level according to a pre-specified fault injection mode; this invention can accurately inject faults to simulate real-world faults, and the implementation of fault injection does not affect the performance of other services, achieving controllable impact and thus ensuring system reliability.

Description

Technical Field

[0001] This invention relates to the field of fault injection technology, and in particular to a method and apparatus for microservice request fault injection based on eBPF. Background Technology

[0002] In reality, various failures can occur at any time, and many are unavoidable, such as sudden power outages. These failures can cause significant losses, therefore we need to improve the system's tolerance to failures and enhance the robustness of the program. To more easily verify the system's reliability and fault tolerance, during the system or application testing phase, we artificially inject faults into the system or application to test how it handles anomalies. Based on the testing results, we can adjust and improve the system or application to reduce abnormal issues that may occur during official use.

[0003] Faults that can occur in microservices include service restarts, service instance termination, and communication failures. The challenge of fault injection for communication failures in a microservices scenario lies in the fact that, using container technology to implement a microservice architecture, services communicate through virtual service addresses in a virtual network. In a distributed environment, there is a mapping relationship between the service addresses of services communicating across hosts and their actual network interface cards (NICs), maintained by the container network interface (CNI) component. Injecting faults into service communication in a microservices architecture allows for observation of microservice behavior in scenarios involving high network latency, packet loss, and connection establishment errors. This enables the improvement of service fault tolerance and stability during network communication failures, significantly enhancing the robustness of application microservices.

[0004] Existing network fault injection tools struggle to obtain the mapping relationship when injecting faults into service communication processes, and their parsing of the source and destination addresses of data packets is not deep enough. They are unable to obtain or modify the data packet content of specific services that need to be injected with faults, making it difficult to implement communication faults in microservice scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a microservice request fault injection method and apparatus based on eBPF, which can accurately inject faults to simulate real-world faults, and the implementation of fault injection does not affect the performance of other services, thus achieving controllable impact and ensuring system reliability.

[0006] To achieve the above objectives, this invention discloses a microservice request fault injection method based on eBPF, which includes the following steps:

[0007] S1. Monitor the process of establishing a socket between the client and the server;

[0008] S2. Capture the socket and obtain the port information of the socket;

[0009] S3. Determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information;

[0010] S4. Based on the judgment result, update the socket as a value to SOCKHASH, and establish an index using the corresponding port information as a key value.

[0011] S5. When the socket in the SOCKHASH calls the sendfile() or sendmsg() function, fault injection is performed at the socket level according to the pre-specified fault injection mode.

[0012] Compared with existing technologies, this invention captures the socket and obtains its port information during the establishment of a socket between the client and the server. It then determines whether the socket is an object to be monitored and injected with faults based on the port information. According to the determination result, the socket is updated as a value in the SOCKHASH, and an index is created using the corresponding port information as a key. This allows for rapid fault injection at the socket level when the socket in the SOCKHASH calls the sendfile() or sendmsg() function, based on a pre-specified fault injection mode. On one hand, this fault injection process does not require modification of application code, the introduction of proxies, or changes to the network transmission path, making it transparent to the user. It is also suitable for finer-grained fault injection at the service level, injecting faults into specific services without affecting the performance of other services, and can perform error code injection. On the other hand, using eBPF technology reduces the extension of the data packet transmission path compared to introducing proxies, minimizing the impact on service latency. Furthermore, it performs fault injection directly in the kernel, making it more efficient.

[0013] Preferably, the fault injection modes include a delay fault mode, a packet loss fault mode, and an error code return fault mode.

[0014] Preferably, fault injection is performed at the socket level using a delayed fault mode, including the following steps:

[0015] During the socket data transmission process, a custom bpf_sys_sleep helper function is used to control the socket transmission action to be delayed by a preset time before data transmission.

[0016] Preferably, fault injection is performed at the socket level using a packet loss fault mode, including the following steps:

[0017] When the client sends a request data packet, packets that meet the requirements are discarded directly at the socket level.

[0018] Preferably, fault injection is performed at the socket level using an error code-returning fault mode, including the following steps:

[0019] When the client sends a request data packet, the HTTP response code of the original data in the response packet buffer at the socket level is modified to change the normal HTTP response code into a 404 error response code.

[0020] Specifically, the HTTP response code is modified byte by byte at the socket level, modifying the original data of the response packet within the buffer.

[0021] Preferably, the socket port information includes the socket's source address port information or destination address port information.

[0022] Preferably, the step of determining whether a socket is an object to be monitored and fault injected based on the socket's port information specifically includes:

[0023] If the destination address port information of the socket is a service address that needs to be injected with a fault, or the source address port information is a service address that needs to be injected with a fault, then the socket is determined to be an object that needs to be monitored and fault injected.

[0024] Preferably, the fault injection at the socket level according to the pre-specified fault injection mode specifically includes:

[0025] If the destination address port information of the socket is the service address that needs to be injected with fault, then fault injection is performed at the socket level in either a delayed fault mode or a packet loss fault mode.

[0026] If the source address port information of the socket is the service address that needs to be injected with a fault, then the fault injection is performed at the socket level in error code return mode.

[0027] Accordingly, the present invention also discloses a microservice request fault injection device based on eBPF, which includes:

[0028] The monitoring module is configured to monitor the process of establishing a socket between the client and the server;

[0029] The capture module is configured to capture sockets and obtain port information of the sockets;

[0030] The judgment module is configured to determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information;

[0031] The first execution module is configured to update the socket as a value to SOCKHASH based on the judgment result, and to establish an index using the corresponding port information as a key value.

[0032] The second execution module is configured to perform fault injection at the socket level according to a pre-specified fault injection mode when the socket in the SOCKHASH calls the sendfile() or sendmsg() function. Attached Figure Description

[0033] Figure 1 This is a flowchart of the microservice request fault injection method based on eBPF of the present invention;

[0034] Figure 2 This is an example diagram showing the service response time and request transmission time when the service responds normally without a service delay fault, as presented in this invention.

[0035] Figure 3 This is an example diagram showing the service response time and request transmission time after a service injection delay failure according to the present invention;

[0036] Figure 4 This is an example diagram showing the service response time and request transmission time when the service responds normally without injecting packet loss faults according to the present invention;

[0037] Figure 5 This is an example diagram showing the service response time and request transmission time after a service injection packet loss failure according to the present invention;

[0038] Figure 6 This is an example diagram of the service response content when the Detail service responds normally when the service returns a failure due to a missing error code.

[0039] Figure 7 This is an example diagram of the service response content of the Detail service after a service injection error code returns a failure according to the present invention;

[0040] Figure 8 This is an example diagram of the service response content when the Ratings service responds normally in the event that the service returned a failure due to a missing error code.

[0041] Figure 9 This is an example diagram of the service response content of the Ratings service after a service injection error code returns a failure according to the present invention;

[0042] Figure 10 This is a structural block diagram of the microservice request fault injection device based on eBPF of the present invention. Detailed Implementation

[0043] To illustrate the technical content, structural features, objectives, and effects of the present invention in detail, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0044] Please see Figure 1As shown, the microservice request fault injection method based on eBPF in this embodiment is suitable for using eBPF technology to implement fault injection on HTTP service request responses in the Linux kernel space. The eBPF technology used in this embodiment is to mount a specific eBPF program at the socket level of the network packet processing process in the Linux kernel space.

[0045] Please see Figures 1-9 As shown, this eBPF-based microservice request fault injection method includes the following steps:

[0046] S1. Monitor the process of establishing a socket between the client and the server.

[0047] The HTTP service request and response process here is as follows:

[0048] When a client requests a service from a server, both parties establish a socket connection through the TCP network protocol's three-way handshake process.

[0049] The client calls the sendfile() or sendmsg() function to send a request data packet to the server;

[0050] The client copies the contents of the request data packet into the socket's buffer, and then the data packet is encapsulated by the TCP / IP protocol stack and forwarded from the client's network card to the server's network card through the physical network or virtual network;

[0051] After the request data packet is received by the server, the server's TCP / IP protocol stack decapsulates the data packet, obtains the content of the request data packet, copies it into the server socket buffer, and then the server application receives and processes the request.

[0052] When the server sends response data packets, it also uses the sendfile() or sendmsg() function, similar to the process of sending and receiving request packets.

[0053] S2. Capture the socket and obtain the port information of the socket.

[0054] Specifically, the socket port information includes the socket's source address port information or destination address port information.

[0055] In this embodiment, a custom SOCK_OPS program is used to monitor and capture the process of establishing a socket between the client and the server.

[0056] S3. Determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information.

[0057] Preferably, if the destination address port information of the socket is a service address that needs to be injected with a fault, or the source address port information is a service address that needs to be injected with a fault, then the socket is determined to be an object that needs to be monitored and fault injected.

[0058] S4. Based on the judgment result, update the socket as a value in SOCKHASH, and establish an index using the corresponding port information as a key value.

[0059] It is understandable that when the socket is an object that needs to be monitored and fault injected, the SOCK_OPS program will update the socket as a value in SOCKHASH, using the corresponding destination address port or source address port as the key value for indexing; when the socket is not an object that needs to be monitored and fault injected, steps S4 and S5 are skipped.

[0060] S5. When the socket in the SOCKHASH calls the sendfile() or sendmsg() function, fault injection is performed at the socket level according to the pre-specified fault injection mode.

[0061] If the destination address port information of the socket is the service address that needs to be injected with fault, then fault injection is performed at the socket level in either a delayed fault mode or a packet loss fault mode.

[0062] If the source address port information of the socket is the service address that needs to be injected with a fault, then the fault injection is performed at the socket level in error code return mode.

[0063] It should be noted that the fault injection in this embodiment is implemented through a custom SK_MSG program.

[0064] Preferably, the fault injection modes include a delay fault mode, a packet loss fault mode, and an error code return fault mode.

[0065] Preferably, fault injection is performed at the socket level using a delayed fault mode, including the following steps:

[0066] During the socket data transmission process, a custom bpf_sys_sleep helper function is used to control the socket transmission action to delay for a preset time before sending data. In this way, the bpf_sys_sleep helper function controls the socket transmission action to sleep for a period of time before continuing to send, thereby achieving the purpose of delay.

[0067] Preferably, fault injection is performed at the socket level using a packet loss fault mode, including the following steps:

[0068] When the client sends a request data packet, the data packet that meets the requirements is directly discarded at the socket level, thus preventing the request data packet from being sent normally. As a result, the client will not receive a response data packet from the server, which manifests as the server not responding to the client's request and a connection error.

[0069] Preferably, fault injection is performed at the socket level using an error code-returning fault mode, including the following steps:

[0070] When the client sends a request data packet, the HTTP response code of the original data in the response packet buffer at the socket level is modified to change the normal HTTP response code into a 404 error response code, which means that the server cannot find the response content and the response fails.

[0071] Specifically, the HTTP response code of the original response packet data in the socket layer buffer is modified byte by byte to ensure the accuracy of fault injection.

[0072] Please see Figures 2-9 As shown below, the results of fault injection under each fault injection mode are presented:

[0073] ① Delayed Fault Mode: By sending a request to the service and observing the change in request latency before and after a 1-second delay fault injection, it can be seen that the service response time is indeed delayed by 1 second, and the shortest and longest request transmission times (Longest transaction and Shortest transaction) are also increased by 1 second. The results of this fault injection show that delayed faults can indeed be injected into specific services. Figure 2 The diagram shows an example of service response time and request transmission time when the service is in normal operation. Figure 3 The diagram illustrates an example of service response time and request transmission time following a service injection latency failure.

[0074] ② Packet loss failure mode: By sending requests to the service and observing the changes in request latency before and after the packet loss failure injection, it can be seen that the service can respond under normal operating conditions, but after the packet loss failure is injected, the service response fails (Permission denied) and an error occurs, which can simulate the failure scenario of the service being inaccessible. Figure 4 This diagram illustrates an example of service response time and request transmission time when the service is functioning normally. Figure 5 The diagram illustrates an example of service response time and request transmission time after a service injection packet loss failure.

[0075] ③ Packet loss failure mode: In this failure scenario, we modify the service response code, changing the normal response code to a failure response code, to simulate a scenario where the service can be accessed but the response is incorrect.

[0076] (1) We perform error code injection in the Detail service: we modify the returned normal response code to 404. As you can see, Figure 6 As shown, under normal circumstances, the Detail service can return detailed information about the book. Figure 7 As shown, after the injection error code returns a failure, the Detail information cannot be obtained;

[0077] (2) We performed error code injection on the Ratings service: we modified the returned normal response code to 404. As you can see, Figure 8 As shown, under normal circumstances, the Ratings service can return the star rating of a book. Figure 9 As shown, after the injection error code returns a failure, the Ratings information cannot be obtained or displayed.

[0078] Please see Figure 10 As shown, correspondingly, the present invention also discloses a microservice request fault injection device based on eBPF, which includes:

[0079] Monitoring module 10 is configured to monitor the process of establishing a socket between the client and the server;

[0080] The capture module 20 is configured to capture sockets and obtain port information of the sockets;

[0081] The judgment module 30 is configured to determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information;

[0082] The first execution module 40 is configured to update the socket as a value to SOCKHASH based on the judgment result, and to establish an index using the corresponding port information as a key value.

[0083] The second execution module 50 is configured to perform fault injection at the socket level according to a pre-specified fault injection mode when the socket in the SOCKHASH calls the sendfile() or sendmsg() function.

[0084] Combination Figures 1-10This invention captures the socket and obtains its port information during the establishment of a socket between the client and server. It then determines whether the socket is an object to be monitored and injected with faults based on the port information. According to the determination result, the socket is updated as a value in the SOCKHASH, and an index is created using the corresponding port information as a key. This allows for rapid fault injection at the socket level when the socket in the SOCKHASH calls the sendfile() or sendmsg() function, based on a pre-specified fault injection mode. On one hand, this fault injection process does not require modification of application code, the introduction of proxies, or changes to the network transmission path, making it transparent to the user. It is also suitable for finer-grained fault injection at the service level, injecting faults into specific services without affecting the performance of other services, and can perform error code injection. On the other hand, using eBPF technology reduces the extension of the data packet transmission path compared to introducing proxies, minimizing the impact on service latency, and performing fault injection directly in the kernel, making it more efficient.

[0085] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A microservice request fault injection method based on eBPF, characterized in that, Includes the following steps: Monitor the process of establishing a socket between the client and the server; Capture the socket and obtain the port information of the socket; Determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information; Based on the judgment result, the socket is updated to SOCKHASH as a value, and an index is established using the corresponding port information as a key value; When a socket in the SOCKHASH calls the sendfile() or sendmsg() function, fault injection is performed at the socket level according to the pre-specified fault injection mode. The fault injection modes include delay fault mode, packet loss fault mode, and error code return fault mode; The socket port information includes the socket's source address port information or destination address port information; The step of determining whether a socket is an object to be monitored and injected with faults based on its port information specifically includes: If the destination address port information of the socket is a service address that needs to be injected with faults, or the source address port information is a service address that needs to be injected with faults, then the socket is determined to be an object that needs to be monitored and fault injected. The fault injection at the socket level according to the pre-specified fault injection mode specifically includes: If the destination address port information of the socket is the service address that needs to be injected with fault, then fault injection is performed at the socket level in either a delayed fault mode or a packet loss fault mode. If the source address port information of the socket is the service address that needs to be injected with a fault, then the fault injection is performed at the socket level in error code return mode.

2. The microservice request fault injection method based on eBPF as described in claim 1, characterized in that, Fault injection at the socket level using a delayed fault mode includes the following steps: During the socket data transmission process, a custom bpf_sys_sleep helper function is used to control the socket transmission action to be delayed by a preset time before data transmission.

3. The microservice request fault injection method based on eBPF as described in claim 1, characterized in that, Fault injection at the socket level using packet loss fault mode includes the following steps: When the client sends a request data packet, packets that meet the requirements are discarded directly at the socket level.

4. The microservice request fault injection method based on eBPF as described in claim 1, characterized in that, Fault injection at the socket level, returning fault modes with error codes, includes the following steps: When the client sends a request data packet, the HTTP response code of the original data in the response packet buffer at the socket level is modified to change the normal HTTP response code into a 404 error response code.

5. The microservice request fault injection method based on eBPF as described in claim 4, characterized in that, Modify the HTTP response code of the original data in the response packet buffer at the socket level, byte by byte.

6. A microservice request fault injection device based on eBPF, characterized in that, include: The monitoring module is configured to monitor the process of establishing a socket between the client and the server; The capture module is configured to capture sockets and obtain port information of the sockets; The judgment module is configured to determine whether the socket is an object that needs to be monitored and fault injected based on the socket's port information; The first execution module is configured to update the socket as a value to SOCKHASH based on the judgment result, and to establish an index using the corresponding port information as a key value. The second execution module is configured to perform fault injection at the socket level according to a pre-specified fault injection mode when the socket in the SOCKHASH calls the sendfile() or sendmsg() function. The fault injection modes include delay fault mode, packet loss fault mode, and error code return fault mode; The socket port information includes the socket's source address port information or destination address port information; The step of determining whether a socket is an object to be monitored and injected with faults based on its port information specifically includes: If the destination address port information of the socket is a service address that needs to be injected with faults, or the source address port information is a service address that needs to be injected with faults, then the socket is determined to be an object that needs to be monitored and fault injected. The fault injection at the socket level according to the pre-specified fault injection mode specifically includes: If the destination address port information of the socket is the service address that needs to be injected with fault, then fault injection is performed at the socket level in either a delayed fault mode or a packet loss fault mode. If the source address port information of the socket is the service address that needs to be injected with a fault, then the fault injection is performed at the socket level in error code return mode.

Images