Method and device for monitoring and analyzing performance of micro-service sidecar based on eBPF

Abstract

The application discloses a micro-service Sidecar performance monitoring and analysis method and device based on eBPF, and belongs to the field of software performance evaluation, which is used for evaluating and analyzing the performance of a Sidecar container in a Kubernetes Pod, and comprises metadata acquisition, Socket event collection, Socket event analysis, network protocol stack event collection, network protocol stack event analysis process, and finally, performance data is sent to a visual component for display and analysis. The application builds a comprehensive index monitoring method based on the Sidecar in the micro-service scenario, can capture the resource consumption of the key steps in the process of the network data packet through the Sidecar container, obtain kernel-level fine-grained monitoring indexes, and makes the performance analysis result of the Sidecar more accurate.

Description

Technical Field

[0001] The present invention relates to the field of microservice software performance evaluation, and in particular to an eBPF-based microservice Sidecar performance monitoring and analysis method and apparatus. Background Art

[0002] With the continuous popularization of cloud-native applications, microservice architecture has become a preferred solution for building high scalability, elasticity and flexibility. However, traffic governance technologies matched with microservice architecture also face many challenges. To address these challenges, functions related to network traffic such as authentication, authorization, service discovery and service governance are often encapsulated into a Sidecar, so as to fully take over the uplink and downlink network request traffic of microservice applications. However, due to the introduction of a new component, Sidecar inevitably brings performance loss, especially when the number of requests per second is extremely high, the performance indicators of Sidecar may even become an important bottleneck restricting microservices. In addition, to maintain the non-invasiveness and light weight of Sidecar, it is difficult to perform data capture and exposure inside Sidecar. Therefore, comprehensive monitoring in kernel mode and user mode based on eBPF tools can achieve the acquisition of various fine-grained key indicators related to network between Sidecar and service containers. Careful observation and analysis of key performance events of Sidecar helps to discover performance bottlenecks, thereby achieving targeted performance optimization. Summary of the Invention

[0003] The purpose of the present invention is to provide an eBPF-based microservice Sidecar performance monitoring and analysis method and apparatus aiming at the deficiencies of the prior art. The present invention can overcome the problems of coarse granularity and high resource occupation of traditional intrusive monitoring technologies, accurately analyze the performance bottleneck of Sidecar, and provide data support for performance optimization.

[0004] The purpose of the present invention is achieved through the following technical solutions: a first aspect of an embodiment of the present invention provides an eBPF-based microservice Sidecar performance monitoring and analysis method, which is applied to a Kubernetes cluster node device deployed with a Sidecar container, and the method includes the following steps:

[0005] (1) Acquiring metadata of Pods in a Kubernetes cluster, and parsing the metadata of the Pods to obtain first related data of the Pod where the Sidecar container is located;

[0006] (2) Monitor the kernel socket events of the Sidecar container and the business container at key monitoring points, filter based on the first relevant data obtained in step (1), capture and parse the connect and accept events of network requests to obtain the second relevant data of the kernel socket;

[0007] (3) Based on the second related data obtained in step (2), the kernel socket events in step (2) are paired to obtain all accept events and connect events inside the Pod when the Pod accepts each network request, and based on this, the transit latency of the Sidecar container and the connection latency between the Sidecar container and the business container are calculated according to the second related data.

[0008] (4) Monitor key kernel points on the Linux kernel network protocol stack, monitor kernel network protocol stack events, and filter based on the first relevant data obtained in step (1) to capture and parse kernel network protocol stack events in order to obtain skb data and event occurrence time of kernel network protocol stack events.

[0009] (5) Based on the skb data and event occurrence time obtained in step (4), the kernel network protocol stack events in step (4) are classified and summarized to obtain the kernel network protocol stack events related to the Pod when the Pod accepts each network request, and based on this, the time consumption between two kernel network protocol stack events, the total time consumption in any connection state, and the container user space processing time are calculated according to the event occurrence time.

[0010] (6) The relay latency of the Sidecar container and the connection latency between the Sidecar container and the service container obtained in step (3), as well as the time consumption between the two kernel network protocol stack events obtained in step (5), the total time consumption in any connection state, and the container user space processing time are transmitted to the Prometheus platform through indicator exposure and collection methods, and visualized through the Grafana platform; and the processing flow of each layer in the kernel network protocol stack is visualized through the Jaeger platform.

[0011] Furthermore, the metadata of the Pod includes the Pod's basic information, configuration information, and status information;

[0012] The first relevant data includes the Pod's IP address, the process ID of the Sidecar container and its child processes, and the process ID of the business container and its child processes.

[0013] Furthermore, the key monitoring points include the Kretprobe point of kprobe:inet_csk_accept, the Kretprobe point of kprobe:tcp_v4_connect, and the Kretprobe and Kprobe points of kprobe:tcp_v6_connect;

[0014] The second relevant data includes the five-tuple of the network request and the process ID, process name, and event occurrence time of the process that triggered the event. The five-tuple of the network request includes the source IP, destination IP, source port number, destination port number, and communication protocol version. The triggering event is either a connect event or an accept event.

[0015] Furthermore, step (3) specifically includes:

[0016] Based on the five-tuple of the network request in the second relevant data and the process ID of the process that triggered the event, all kernel socket events in a network request are paired: using the process ID of the process of the Sidecar container's accept event and the process ID of the process of the Sidecar container's connect event, the accept event of the Sidecar container and the connect event of the Sidecar container are paired to obtain the first set of paired kernel socket events; using the source IP and port number in the connect event of the Sidecar container and the source IP and port number in the accept event of the business container, the connect event of the Sidecar container and the accept event of the business container are paired to obtain the second set of paired kernel socket events.

[0017] The first set of paired kernel socket events and the second set of paired kernel socket events are combined according to the common Sidecar container's connect event to obtain all the accept and connect events in a complete network request.

[0018] The transit latency of the Sidecar container is calculated based on the occurrence times of the accept and connect events of the paired Sidecar containers; the connection latency between the Sidecar container and the business container is calculated based on the occurrence times of the connect event of the paired Sidecar container and the accept event of the business container.

[0019] Furthermore, the key kernel points include the napi_gro_receive point, __dev_queue_xmit point, netif_rx point, __netif_receive_skb point, ip_rcv point, ip_rcv_finish point, ip_output point, and ip_finish_output point;

[0020] The skb data includes the kernel network protocol stack event quintuple, IP layer protocol, TCP flags, current network interface card and its name, CPU number, destination MAC address, and IP datagram length; the kernel network protocol stack event quintuple includes the source IP address, destination IP address, source port number, destination port number, and communication protocol version.

[0021] Furthermore, step (5) includes the following sub-steps:

[0022] (5.1) Based on the five-tuple of kernel network protocol stack events in the skb data obtained in step (4), determine the transmission direction of the current kernel network protocol stack event;

[0023] (5.2) Obtain the connection status of the current kernel network protocol stack event based on the IP layer protocol and TCP flags in the skb data obtained in step (4);

[0024] (5.3) Use the source port number and destination port number of the kernel network protocol stack event to identify the connection state of the current kernel network protocol stack event, so as to generate the identifier ID of the current connection state;

[0025] (5.4) Combine kernel network protocol stack events according to the identification ID of the connection state, and group kernel network protocol stack events with the same identification ID: According to the transmission direction of the current kernel network protocol stack event obtained in step (5.1) and the connection state of the current kernel network protocol stack event determined in step (5.2), classify and store all kernel network protocol stack events, and sort the kernel network protocol stack events in each category according to the event occurrence time obtained in step (4) to obtain the sorted kernel network protocol stack events related to the Pod when the Pod accepts each network request;

[0026] (5.5) Based on the sorted kernel network protocol stack events obtained in step (5.4), calculate the time consumption between the two kernel network protocol stack events according to the event occurrence time of the two kernel network protocol stack events; calculate the total time consumption of the connection state according to the earliest event occurrence time and the latest event occurrence time of any connection state; calculate the container user space processing time according to the event occurrence time corresponding to the transmission direction "from the Sidecar container to the business container" and the event occurrence time corresponding to "from the business container to the Sidecar container".

[0027] Furthermore, there are six types of transmission directions for the kernel network protocol stack events: ① from the business container to the sidecar container, ② from the sidecar container to the business container, ③ from the Pod to the remote caller, ④ from the remote caller to the Pod, ⑤ from the local node address to the remote caller, and ⑥ from the remote caller to the local node address.

[0028] The specific steps (5.1) are as follows: Based on the five-tuple of the kernel network protocol stack event in the skb data obtained in step (4), the following judgments are made: If the source IP address of the kernel network protocol stack event is localhost and the destination IP address is also localhost, then the kernel network protocol stack event is an internal network event of the Pod. Then, based on the source port number and the destination port number, the following judgments are made: If the source port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "① from the business container to the Sidecar container"; if the destination port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "② from the Sidecar container to the business container"; if the source IP address of the kernel network protocol stack event is the same as the Pod IP, then the following judgments are made based on the source port number: If the source port number of the kernel network protocol stack event is the same as the port number of the Sidecar container, then the transmission direction of the kernel network protocol stack event is "② from the Sidecar container to the business container". If the port number of the kernel network protocol stack event is the same as the port number of the business container, then the transmission direction of the kernel network protocol stack event is "③ from Pod to remote caller"; if the destination IP address of the kernel network protocol stack event is the same as the Pod IP, then the destination port number is used for further judgment: if the destination port number of the kernel network protocol stack event is the same as the port number of the Sidecar container or the business container, then the transmission direction of the kernel network protocol stack event is "④ from remote caller to Pod"; if the source IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑤ from local node address to remote caller"; if the destination IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑥ from remote caller to local node address"; if none of the above rules are matched, then the kernel network protocol stack event is discarded.

[0029] Furthermore, the connection status of the kernel network protocol stack events includes establishing a connection, sending data, closing a connection, and acknowledging receipt of data packets.

[0030] A second aspect of this invention provides a microservice sidecar performance monitoring and analysis device based on eBPF, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program to implement the above-described microservice sidecar performance monitoring and analysis method based on eBPF.

[0031] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described eBPF-based microservice Sidecar performance monitoring and analysis method.

[0032] The beneficial effects of this invention are that it can analyze the performance of Sidecar relay and connection establishment based on socket events in the kernel in a non-intrusive and low-resource-consumption manner; it can also model and track network requests between the Sidecar container and the business container inside the Pod in a non-intrusive way within the kernel network protocol stack. It can not only analyze the time consumption of network requests before and after key kernel points, but also extract and display the network status at key points, enabling timely and accurate detection of network bottlenecks and abnormal resource consumption in the Pod. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the relationship between the various monitoring components in this invention;

[0034] Figure 2 This is a schematic diagram of the monitoring results of key performance indicators in this invention;

[0035] Figure 3 This is a schematic diagram of the monitoring results of the network protocol stack invocation process in this invention;

[0036] Figure 4 This is a schematic diagram of a microservice Sidecar performance monitoring and analysis device based on eBPF in this invention. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this application.

[0038] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0039] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0040] The eBPF-based microservice Sidecar performance monitoring and analysis method of this invention is applied to Kubernetes cluster node devices with deployed Sidecar containers, and specifically includes the following steps:

[0041] (1) Obtain the metadata of the Pod in the Kubernetes cluster and parse the metadata of the Pod to obtain the first relevant data of the Pod where the Sidecar container is located.

[0042] Furthermore, the metadata of a Pod includes basic information, configuration information, and status information of the Pod.

[0043] Furthermore, the first relevant data includes the Pod's IP address, the process ID of the Sidecar container and its child processes, and the process ID of the business container and its child processes.

[0044] Specifically, first, the kubeconfig file is loaded to specify the cluster configuration file. Then, using the Go library of the Kubernetes cluster, a GET request is made to the Kubernetes cluster API to obtain the metadata of the Pod to be monitored. The Pod metadata includes basic Pod information (metadata), Pod configuration information (spec), and Pod status information (status). Next, the Pod metadata is parsed to extract the current node's IP, PodIP, and Pod container information. The Pod container information includes the image name, container ID, and container port. Then, the Sidecar container and the application container are determined based on the image name. Finally, the Docker API is requested using the Go library of Docker to obtain the processes and their child processes inside the Sidecar container and the application container using the container ID. This yields the process IDs of the Sidecar container process and its child processes, as well as the process IDs of the application container process and its child processes.

[0045] (2) Monitor the kernel socket events of the Sidecar container and the business container at key monitoring points, filter based on the first relevant data obtained in step (1), capture and parse the connect and accept events of network requests to obtain the second relevant data of the kernel socket.

[0046] Furthermore, key monitoring points include the Kretprobe point of kprobe:inet_csk_accept, the Kretprobe point of kprobe:tcp_v4_connect, and the Kretprobe and Kprobe points of kprobe:tcp_v6_connect. In operating systems compatible with the technology described herein, all of these monitoring points can be searched using the bpftrace tool. Taking kprobe:inet_csk_accept as an example, it can be retrieved and confirmed using the command "bpftrace -l kprobe:inet_csk_accept".

[0047] It should be noted that kernel sockets refer to events related to sockets in the kernel, namely the connect event and the accept event.

[0048] Furthermore, the second relevant data includes the five-tuple of the network request and the process ID, process name, and event occurrence time of the process that triggered the event; wherein, the five-tuple of the network request includes the source IP, destination IP, source port number, destination port number, and communication protocol version, and the triggering event is either a connect event or an accept event.

[0049] Specifically, monitor the kernel socket events of the Sidecar container and the business container at key monitoring points, such as... Figure 1 As shown, the `connect` and `accept` events during the Sidecar process of a network request are captured. At their respective kernel monitoring points, the data in the CPU registers at that moment is obtained, and the return value of this kernel function is retrieved from the RC register. The pointer to the Linux kernel `sock` structure is then parsed. Subsequently, based on the structure's definition in the kernel, the corresponding fields are retrieved using offsets. Since this structure may change with different kernel versions, the data structure corresponding to the kernel version must be used as the definition when compiling the eBPF program. Next, the source IP address, destination IP address, source port number, destination port number, and communication protocol version of this network request are parsed, and the PID and TGID of the process that triggered the event are obtained through eBPF's helper functions. Here, based on the first relevant data such as the PID of the container process obtained in step (1), irrelevant events in the system are filtered out. Finally, all accept and connect events of the business container and the sidecar container are obtained. These obtained events include the process ID and process name of the process corresponding to the accept or connect event, as well as the event occurrence time. The obtained data and the event occurrence time are packaged together and sent to the user-space monitoring program through eBPF's perf map. The user-space monitoring program puts the obtained events into a message queue for further analysis and processing in subsequent steps.

[0050] It should be understood that the current triggering event could be an accept event or a connect event. If the current triggering event is an accept event, the second relevant data obtained includes the process ID, process name, and the event occurrence time of the process that triggered the accept event. PID and TGID are the kernel-level PID (i.e., the process ID of the process that triggered the event) and the user-level PID, respectively.

[0051] (3) Based on the second related data obtained in step (2), the kernel socket events in step (2) are paired to obtain all accept events and connect events inside the Pod when the Pod accepts each network request, and based on this, the transit latency of the Sidecar container and the connection latency between the Sidecar container and the business container are calculated according to the second related data.

[0052] Specifically, Socket events are reassembled and analyzed, reassembling them according to a complete network service request. The event message queue from the previous steps is read, and the `accept` event of the Sidecar container in a single request is paired with the `connect` event. Similarly, the `connect` event of the Sidecar container in a single request is paired with the `accept` event of the business container. This allows obtaining the user-space processing latency (i.e., relay latency) from when the Sidecar receives an external request call to when it establishes a connection with the business container, as well as the connection establishment latency from when the Sidecar establishes a connection with the business container and the latency when the connection is established. The specific pairing method varies depending on the event type, but it's important to note that the data must first be sorted by time. For the first pairing, the common PID and TGID information from both events is used; that is, if two events from the Sidecar container share a common PID and TGID, they are considered compatible; otherwise, the event is discarded. For the second pairing, the source IP and port number from the `connect` event of the Sidecar container are used to pair with the source IP and port number from the `accept` event of the business container. The two paired events are then combined based on the shared connect event of the Sidecar container, integrating all accept and connect events in a complete request process, serving as the basis for user-space and kernel-space monitoring and analysis. Based on this, the user-space processing latency of the Sidecar container and the TCP connection establishment latency between the Sidecar container and the application container are calculated.

[0053] It should be understood that the second relevant data obtained in step (2) includes the event occurrence time of the triggering event. All accept and connect events in a network request process after pairing are completed also have corresponding event occurrence times. The TCP connection establishment delay between the Sidecar container and the business container can be calculated based on the event occurrence time of the connect event of the paired Sidecar container and the event occurrence time of the accept event of the business container. The user-space processing delay of the Sidecar container, i.e., the relay delay of the Sidecar container, can be calculated based on the event occurrence time of the connect event of the paired Sidecar container and the event occurrence time of the accept event of the Sidecar container.

[0054] (4) Monitor key kernel points on the Linux kernel network protocol stack, monitor kernel network protocol stack events, and filter based on the first relevant data obtained in step (1) to capture and parse kernel network protocol stack events in order to obtain the skb data and event occurrence time of kernel network protocol stack events.

[0055] Furthermore, key kernel points include, but are not limited to, the napi_gro_receive point, the __dev_queue_xmit point, the netif_rx point, the __netif_receive_skb point, the ip_rcv point, the ip_rcv_finish point, the ip_output point, and the ip_finish_output point. The method for obtaining key kernel points is the same as in step (2).

[0056] Furthermore, the skb data includes the kernel network protocol stack event quintuple, IP layer protocol, TCP flags, current network interface card and its name, CPU number, destination MAC address, and IP datagram length; among which, the kernel network protocol stack event quintuple includes source IP address, destination IP address, source port number, destination port number, and communication protocol version.

[0057] Specifically, key kernel points on the Linux kernel network protocol stack are monitored to capture kernel network protocol stack events. For example... Figure 1As shown, the purpose of this step is to monitor the relevant information of various key kernel points in the kernel network protocol stack for network requests, discover the event differences between two points, and then analyze the intermediate kernel processing to find special changes or abnormal states. Monitor each key kernel point, obtain the data on the CPU register at this time, and parse the pointer of the kernel data structure sk_buff (i.e., skb). After that, the parsing method of the structure is the same as the method in step (2). From the skb structure, the skb data of the kernel network protocol stack event can be obtained, which includes several event parameters. At the same time, the event occurrence time of the current kernel network protocol stack event, the pid and tgid of the current process, the process name, the current network namespace and other related information are obtained through the eBPF helper function. It is worth noting that during system testing, there is often a phenomenon where the process corresponding to the process number does not match the location of the network event. Therefore, the Pod information obtained in step (1) is used for filtering in the event filtering process, that is, the container process number plus PodIP is used to filter the event, and PodIP is used to filter both the source IP and the destination IP. To ensure system performance, process IDs and PodIPs should be filtered as early as possible. Finally, the encapsulated event data is packaged and sent to the user-space monitoring program via eBPF's perf map. The user-space monitoring program then passes the acquired events to a message queue for further analysis and processing.

[0058] (5) Based on the skb data and event occurrence time obtained in step (4), the kernel network protocol stack events in step (4) are classified and summarized to obtain the kernel network protocol stack events related to the Pod when the Pod accepts each network request. Based on this, the time consumption between two kernel network protocol stack events, the total time consumption in any connection state, and the container user space processing time are calculated according to the event occurrence time.

[0059] In this embodiment, the kernel network protocol stack events are analyzed and reassembled according to a complete service request. The event message queue from step (4) is read, and the data transmission process in the network protocol stack is combined from top to bottom. Considering that there is an overlay network in the Kubernetes cluster, where IP addresses, port numbers, and network namespaces will change, a more complex combination method is used here, as shown below.

[0060] (5.1) Based on the quintuple of kernel network protocol stack events in the skb data obtained in step (4), determine the transmission direction of the current kernel network protocol stack event.

[0061] Furthermore, there are six types of transmission directions for kernel network protocol stack events: ① from business container to sidecar container, ② from sidecar container to business container, ③ from Pod to remote caller, ④ from remote caller to Pod, ⑤ from local node address to remote caller, and ⑥ from remote caller to local node address.

[0062] Specifically, based on the 5-tuple of the kernel network protocol stack event in the skb data obtained in step (4), the following judgments are made: If the source IP address of the kernel network protocol stack event is localhost and the destination IP address is also localhost, then the kernel network protocol stack event is an internal network event of the Pod. Further judgments are made based on the source and destination port numbers: If the source port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "① from the business container to the Sidecar container"; if the destination port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "② from the Sidecar container to the business container"; if the source IP address of the kernel network protocol stack event is the same as the Pod IP, then the following judgments are made based on the source port number: If the source port number of the kernel network protocol stack event is the same as the port number of the Sidecar container or the business container... If the port number of the kernel network protocol stack event is the same as that of the Pod, then the transmission direction of the kernel network protocol stack event is "③ from Pod to remote caller"; if the destination IP address of the kernel network protocol stack event is the same as that of the Pod IP, then the destination port number is used for further judgment: if the destination port number of the kernel network protocol stack event is the same as that of the Sidecar container or the application container, then the transmission direction of the kernel network protocol stack event is "④ from remote caller to Pod"; if the source IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑤ from local node address to remote caller"; if the destination IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑥ from remote caller to local node address"; if none of the above rules are matched, then the kernel network protocol stack event is discarded.

[0063] It should be understood that localhost refers to the local IP address, which is generally 127.0.0.1, but 127.0.0.6 is also sometimes used.

[0064] (5.2) Obtain the connection status of the current kernel network protocol stack event based on the IP layer protocol and TCP flags in the skb data obtained in step (4).

[0065] Furthermore, the connection status of kernel network protocol stack events includes establishing a connection, sending data, closing a connection, and acknowledging receipt of data packets.

[0066] Specifically, the IP layer protocol generally only includes TCP and UDP. UDP is stateless, only recording data without any additional processing. Since the IP layer protocol is TCP, the TCP flags field needs to be parsed. These TCP flags include the TCP status bits SYN, PSH, FIN, and ACK, such as... Figure 1 As shown, it can identify the current state of the connection.

[0067] (5.3) Use the source port number and destination port number of the kernel network protocol stack event to identify the connection status of the current kernel network protocol stack event, so as to generate the identifier ID of the current connection status.

[0068] Specifically, an identifier ID for the current connection is generated, using the port number from the kernel network protocol stack event. Since Network Address Translation (NAT) is not used in the Pod network, the source port number at the time of the request can uniquely identify a connection. Because bidirectional network transmission exists, the Sidecar container's port number is used to help determine which port number in the network event is the source port number of the TCP connection—that is, the network event port number that is not the Sidecar container's port number. Because the Pod uses the same network namespace, port numbers will not be duplicated. Therefore, if the source port number is the Sidecar container's port number, it indicates a connection from the Sidecar container to the application container; if the destination port number is the Sidecar container's port number, it indicates a connection from the application container to the Sidecar container.

[0069] It should be understood that NAT is a protocol that translates the IP address in the IP packet header into another IP address. Some network requests in a Kubernetes cluster are translated by iptables NAT, but not inside a Pod, meaning that the IP address and port remain unchanged.

[0070] (5.4) Combine kernel network protocol stack events based on the connection state identifier ID. For kernel network protocol stack events with the same identifier ID, group and record them as follows: Based on the transmission direction of the current kernel network protocol stack event obtained in step (5.1) and the connection state of the current kernel network protocol stack event determined in step (5.2), classify and store all kernel network protocol stack events, and sort the kernel network protocol stack events under each category according to the event occurrence time obtained in step (4) to obtain the sorted kernel network protocol stack events related to the Pod when the Pod accepts each network request. At this time, all the kernel network protocol stack event information of interest for a network request is obtained. The sorted and classified kernel network protocol stack events can be used directly for event analysis in subsequent steps.

[0071] (5.5) Based on the sorted kernel network protocol stack events obtained in step (5.4), calculate the time consumption between the two kernel network protocol stack events according to the event occurrence time of the two kernel network protocol stack events; calculate the total time consumption of the connection state according to the earliest event occurrence time and the latest event occurrence time of any connection state; calculate the container user space processing time according to the event occurrence time corresponding to the transmission direction "from the Sidecar container to the business container" and the event occurrence time corresponding to the business container to the Sidecar container.

[0072] Specifically, based on the sorted kernel network protocol stack events obtained in step (5.4), the time consumption between the two kernel network protocol stack events can be calculated according to their occurrence times. The total time consumption for any connection state is calculated based on the earliest and latest occurrence times of the events. For example, when the connection state is closed, there are corresponding sorted kernel network protocol stack events. The time difference can be calculated based on the occurrence times of the first event (i.e., the earliest event) and the last event (i.e., the latest event), and this time difference is the total time consumption for that connection state. After the connection is established, the Sidecar sends data to the service container (PSH state), with the corresponding transmission direction being ② from the Sidecar container to the service container. The service container then starts sending data to the Sidecar (PSH state), with the corresponding transmission direction being ① from the service container to the Sidecar container. The container user-space processing time can be obtained by calculating the time difference between the occurrence times of the two events; for example, the service container processing time is... Figure 3 The time difference between the fifth and fourth lines from the bottom of the pipeline; the time difference between the sixth and thirteenth lines is the user-mode transfer time difference of the Sidecar container.

[0073] (6) The relay latency of the Sidecar container and the connection latency between the Sidecar container and the business container obtained in step (3), as well as the time consumption between the two kernel network protocol stack events obtained in step (5), the total time consumption in any connection state, and the container user space processing time are transmitted to the Prometheus platform through indicator exposure and collection methods, and visualized through the Grafana platform; and the processing flow of each layer in the kernel network protocol stack is visualized through Jaeger.

[0074] Specifically, the relay latency of the Sidecar container and the connection establishment latency between the Sidecar container and the business container calculated in step (3), as well as the time consumption between the two kernel network protocol stack events calculated in step (5), the total time consumption in any connection state, and the container user-space processing time are displayed and analyzed. The Prometheus exporter is used to analyze the user-space processing latency of the Sidecar container and the TCP connection establishment latency between the Sidecar container and the business container in step (3), and the percentile distribution is calculated using the Prometheus SummaryVec tool. In addition, the performance information of the host device is obtained through the Prometheus Node Exporter, and container-level performance metrics data, such as the CPU, memory, network, and file system resource usage of the Pod container, are obtained through cAdvisor (ContainerAdvisor). Finally, the above information is reported to Prometheus for storage and displayed using the Grafana platform. The overall dashboard results are as follows: Figure 2 As shown. The kernel network protocol stack event combination obtained in step (5) is analyzed using the OpenTracing protocol. It is divided into several spans according to the states during the TCP connection process and reported to the Jaeger platform for visualization. This allows for analysis of the various states within the TCP connection, as shown in the results. Figure 3 As shown.

[0075] Corresponding to the aforementioned embodiments of the eBPF-based microservice sidecar performance monitoring and analysis method, the present invention also provides embodiments of the eBPF-based microservice sidecar performance monitoring and analysis device.

[0076] See Figure 4 The present invention provides a microservice sidecar performance monitoring and analysis device based on eBPF, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the microservice sidecar performance monitoring and analysis method based on eBPF in the above embodiments.

[0077] The embodiment of the microservice sidecar performance monitoring and analysis device based on eBPF of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities where the eBPF-based microservice Sidecar performance monitoring and analysis device of this invention is located, except for... Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0078] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0079] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the eBPF-based microservice Sidecar performance monitoring and analysis method described in the above embodiments.

[0081] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A microservice sidecar performance monitoring and analysis method based on eBPF, applied to Kubernetes cluster node devices with deployed sidecar containers, characterized in that, The method includes the following steps: (1) Obtain the metadata of the Pod in the Kubernetes cluster and parse the metadata of the Pod to obtain the first relevant data of the Pod where the Sidecar container is located; (2) Monitor the kernel socket events of the Sidecar container and the business container at key monitoring points, filter based on the first relevant data obtained in step (1), capture and parse the connect and accept events of network requests to obtain the second relevant data of the kernel socket; (3) Based on the second related data obtained in step (2), the kernel socket events in step (2) are paired to obtain all accept events and connect events inside the Pod when the Pod accepts each network request, and based on this, the transit latency of the Sidecar container and the connection latency between the Sidecar container and the business container are calculated according to the second related data. (4) Monitor key kernel points on the Linux kernel network protocol stack, monitor kernel network protocol stack events, and filter based on the first relevant data obtained in step (1) to capture and parse kernel network protocol stack events in order to obtain skb data and event occurrence time of kernel network protocol stack events. (5) Based on the skb data and event occurrence time obtained in step (4), the kernel network protocol stack events in step (4) are classified and summarized to obtain the kernel network protocol stack events related to the Pod when the Pod accepts each network request, and based on this, the time consumption between two kernel network protocol stack events, the total time consumption in any connection state, and the container user space processing time are calculated according to the event occurrence time. (6) The relay latency of the Sidecar container and the connection latency between the Sidecar container and the service container obtained in step (3), as well as the time consumption between the two kernel network protocol stack events obtained in step (5), the total time consumption in any connection state, and the container user space processing time are transmitted to the Prometheus platform through indicator exposure and collection methods, and visualized through the Grafana platform; and the processing flow of each layer in the kernel network protocol stack is visualized through the Jaeger platform.

2. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 1, characterized in that, The metadata of the Pod includes the Pod's basic information, configuration information, and status information; The first relevant data includes the Pod's IP address, the process ID of the Sidecar container and its child processes, and the process ID of the business container and its child processes.

3. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 1, characterized in that, The key monitoring points include the Kretprobe point of kprobe:inet_csk_accept, the Kretprobe point of kprobe:tcp_v4_connect, and the Kretprobe and Kprobe points of kprobe:tcp_v6_connect; The second relevant data includes the five-tuple of the network request and the process ID, process name, and event occurrence time of the process that triggered the event. The five-tuple of the network request includes the source IP, destination IP, source port number, destination port number, and communication protocol version. The triggering event is either a connect event or an accept event.

4. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 1, characterized in that, Step (3) specifically includes: Based on the 5-tuple of the network request in the second relevant data and the process ID of the process that triggered the event, all kernel socket events in a network request are paired: using the process ID of the process of the Sidecar container's accept event and the process ID of the process of the Sidecar container's connect event, the accept event of the Sidecar container and the connect event of the Sidecar container are paired to obtain the first set of paired kernel socket events; using the source IP and port number in the connect event of the Sidecar container and the source IP and port number in the accept event of the business container, the connect event of the Sidecar container and the accept event of the business container are paired to obtain the second set of paired kernel socket events. The first set of paired kernel socket events and the second set of paired kernel socket events are combined according to the common Sidecar container's connect event to obtain all the accept and connect events in a complete network request. The transit latency of the Sidecar container is calculated based on the occurrence times of the accept and connect events of the paired Sidecar containers; the connection latency between the Sidecar container and the business container is calculated based on the occurrence times of the connect event of the paired Sidecar container and the accept event of the business container.

5. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 1, characterized in that, The key kernel points include the napi_gro_receive point, __dev_queue_xmit point, netif_rx point, __netif_receive_skb point, ip_rcv point, ip_rcv_finish point, ip_output point, and ip_finish_output point; The skb data includes the kernel network protocol stack event quintuple, IP layer protocol, TCP flags, current network interface card and its name, CPU number, destination MAC address, and IP datagram length; the kernel network protocol stack event quintuple includes the source IP address, destination IP address, source port number, destination port number, and communication protocol version.

6. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 1, characterized in that, Step (5) includes the following sub-steps: (5.1) Based on the five-tuple of kernel network protocol stack events in the skb data obtained in step (4), determine the transmission direction of the current kernel network protocol stack event; (5.2) Obtain the connection status of the current kernel network protocol stack event based on the IP layer protocol and TCP flags in the skb data obtained in step (4); (5.3) Use the source port number and destination port number of the kernel network protocol stack event to identify the connection state of the current kernel network protocol stack event, so as to generate the identifier ID of the current connection state; (5.4) Combine kernel network protocol stack events according to the identification ID of the connection state, and group kernel network protocol stack events with the same identification ID: According to the transmission direction of the current kernel network protocol stack event obtained in step (5.1) and the connection state of the current kernel network protocol stack event determined in step (5.2), classify and store all kernel network protocol stack events, and sort the kernel network protocol stack events in each category according to the event occurrence time obtained in step (4) to obtain the sorted kernel network protocol stack events related to the Pod when the Pod accepts each network request; (5.5) Based on the sorted kernel network protocol stack events obtained in step (5.4), calculate the time consumption between the two kernel network protocol stack events according to the event occurrence time of the two kernel network protocol stack events; calculate the total time consumption of the connection state according to the earliest event occurrence time and the latest event occurrence time of any connection state; calculate the container user space processing time according to the event occurrence time corresponding to the transmission direction "from the Sidecar container to the business container" and the event occurrence time corresponding to the transmission direction "from the business container to the Sidecar container".

7. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 6, characterized in that, There are six types of transmission directions for the kernel network protocol stack events: ① from business container to sidecar container, ② from sidecar container to business container, ③ from Pod to remote caller, ④ from remote caller to Pod, ⑤ from local node address to remote caller, and ⑥ from remote caller to local node address. The specific steps (5.1) are as follows: Based on the five-tuple of the kernel network protocol stack event in the skb data obtained in step (4), the following judgments are made: If the source IP address of the kernel network protocol stack event is localhost and the destination IP address is also localhost, then the kernel network protocol stack event is an internal network event of the Pod. Then, based on the source port number and the destination port number, the following judgments are made: If the source port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "① from the business container to the Sidecar container"; if the destination port number is the port number of the business container, then the transmission direction of the kernel network protocol stack event is "② from the Sidecar container to the business container"; if the source IP address of the kernel network protocol stack event is the same as the Pod IP, then the following judgments are made based on the source port number: If the source port number of the kernel network protocol stack event is the same as the port number of the Sidecar container, then the transmission direction of the kernel network protocol stack event is "② from the Sidecar container to the business container". If the port number of the kernel network protocol stack event is the same as the port number of the business container, then the transmission direction of the kernel network protocol stack event is "③ from Pod to remote caller"; if the destination IP address of the kernel network protocol stack event is the same as the Pod IP, then the destination port number is used for further judgment: if the destination port number of the kernel network protocol stack event is the same as the port number of the Sidecar container or the port number of the business container, then the transmission direction of the kernel network protocol stack event is "④ from remote caller to Pod"; if the source IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑤ from local node address to remote caller"; if the destination IP address of the kernel network protocol stack event is the local node IP, then the transmission direction of the kernel network protocol stack event is "⑥ from remote caller to local node address"; if none of the above rules are matched, then the kernel network protocol stack event is discarded.

8. The microservice sidecar performance monitoring and analysis method based on eBPF according to claim 6, characterized in that, The connection status of the kernel network protocol stack events includes establishing a connection, sending data, closing a connection, and acknowledging receipt of a data packet.

9. A microservice sidecar performance monitoring and analysis device based on eBPF, comprising one or more processors and memory, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program to implement the eBPF-based microservice Sidecar performance monitoring and analysis method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, is used to implement the eBPF-based microservice Sidecar performance monitoring and analysis method as described in any one of claims 1-8.

Images