A container network packet loss monitoring system and method
By deploying layered packet loss collectors and analyzers on container nodes, combined with eBPF and OVS technologies, the performance bottleneck and location challenges of packet loss monitoring in ultra-large-scale container clusters are solved, achieving efficient, low-overhead, multi-dimensional monitoring and improving operational efficiency and reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INESA (GRP) CO LTD
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to achieve full-coverage real-time monitoring in ultra-large-scale container clusters, exhibiting performance bottlenecks and high overhead. They also fail to accurately pinpoint the root cause of packet loss and lack multi-dimensional data analysis, resulting in low operational efficiency.
Deploy a container network packet loss collector on each container node, employ a layered packet capture and processing strategy, combine eBPF and OVS technologies for precise capture and filtering, utilize columnar database storage and a packet loss data analyzer for multi-dimensional analysis, locate packet loss locations and generate alarms.
It achieves efficient and low-overhead packet loss monitoring for container networks, accurately locates packet loss points, reduces data transmission volume, lowers operation and maintenance costs, and improves the reliability and operation and maintenance efficiency of containerized applications.
Smart Images
Figure CN122160294A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information data processing technology, and in particular to a container network packet loss monitoring system and method. Background Technology
[0002] With the rapid development of cloud computing, microservice architecture, and container technology, the deployment and application of large-scale container clusters have become a core pillar of modern enterprise IT infrastructure. Container technology, with its advantages of lightweight design, rapid deployment, and environmental consistency, is widely used in the development and operation of various application systems. However, as the scale of container clusters continues to expand, network performance issues, especially packet loss, are becoming increasingly prominent. This not only seriously affects the service quality and user experience of containerized applications but may also lead to business interruptions and economic losses.
[0003] The industry has accumulated some technical expertise in container network packet loss monitoring. Traditional solutions primarily rely on engineers using packet capture tools such as tcpdump and Wireshark to perform packet capture and deep packet inspection analysis on specific network nodes. To achieve more proactive monitoring, some solutions choose to deploy dedicated network probes in container networks to actively detect network connectivity and packet loss. Furthermore, mainstream container networking solutions like Calico and Flannel also provide built-in monitoring capabilities, enabling the acquisition of basic monitoring data at the overlay network layer and supporting simple fault alerts.
[0004] However, these existing technical solutions have several significant shortcomings when dealing with ultra-large-scale container clusters. Firstly, there are scalability issues. Traditional monitoring solutions struggle to achieve full-coverage real-time monitoring at the scale of tens of thousands of containers. Data collection and storage often face performance bottlenecks, and the monitoring granularity is relatively coarse, failing to meet the needs of refined operations and maintenance. Secondly, system performance overhead is significant. In particular, methods such as packet capture analysis consume substantial CPU and memory resources, making continuous operation in production environments difficult and potentially impacting the normal operation of business containers. For example, the invention disclosed in CN114465897A, which describes a method, apparatus, and system for monitoring data packets in business flows, requires sending identification information of the monitored business flows to each network node to return packet loss information, increasing the burden on the container network.
[0005] In terms of problem localization efficiency, existing solutions often only detect packet loss but cannot quickly and accurately pinpoint the root cause. They lack automated fault diagnosis capabilities, requiring significant manual intervention during troubleshooting. Monitoring real-time performance is also insufficient; most solutions use periodic sampling, making it difficult to capture millisecond-level instantaneous packet loss, resulting in substantial delays in fault alarms. Furthermore, monitoring dimensions are too singular, typically focusing only on network-level packet loss and lacking correlation analysis of multi-dimensional data such as container runtime status and system resources, failing to establish a global performance view. In addition, the value of monitoring data is not fully utilized; the collected data lacks in-depth analysis and mining, no effective problem prediction models have been established, and historical data is insufficient to guide optimization decisions.
[0006] These technical shortcomings mean that operations and maintenance personnel often have to spend a lot of time and manpower dealing with packet loss issues in container networks, which seriously affects the availability and service quality of containerized applications. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a high-performance, low-overhead, multi-dimensional container network packet loss monitoring system and method that can adapt to the operation and maintenance needs of modern cloud-native applications, provide accurate fault location capabilities, and ultimately help enterprises improve the reliability and operation and maintenance efficiency of containerized applications.
[0008] The objective of this invention can be achieved through the following technical solutions: A container network packet loss monitoring system includes: A container network packet loss collector is deployed on each node of a container cluster, which is built on the container cluster. It is used to capture packets dropped by the container network and extract the five-tuple information from the captured packets. The five-tuple information includes the source IP address, destination IP address, source port, destination port and transport protocol. The container network packet loss database stores the five-tuple information of data packets captured by the container network packet loss collector. The packet loss data analyzer is used to determine the microservice components associated with the packets in the container cluster topology information based on the pre-acquired container cluster topology information and the five-tuple information corresponding to each packet stored in the container network packet loss database, locate the location of the dropped packets, and generate alarms.
[0009] Furthermore, the container network packet loss collector includes a packet capture layer and a packet processing layer; The packet capture layer is used to identify the data plane of packets dropped by the container network. Different capture strategies are used to deploy probes for the data plane based on the Linux kernel network protocol stack and the data plane based on OVS, respectively, in order to capture packets dropped by the container network. The packet processing layer is used to filter and extract the five-tuple information of the packets obtained by the packet capture layer, and then forward them to the container network packet loss database.
[0010] Furthermore, the data plane capture strategy based on the Linux kernel network protocol stack includes: Based on eBPF technology, packet loss capture probes are deployed on the ingress and egress queues of critical network nodes in the Linux kernel, and the classifiers of the critical network nodes are modified so that the packet loss capture probes trigger the capture logic before the packets are executed by the filtering rules of the critical network nodes. For the data packets captured by the packet loss capture probe, it is determined whether the data packets are dropped by the container network based on the protocol characteristics and network interface information of the data packets, so as to send them to the data packet processing layer.
[0011] Furthermore, the process of determining whether a data packet is dropped by the container network includes: determining whether the data packet meets either a preset first judgment criterion or a preset second judgment criterion; if it does, it is considered a data packet dropped by the container network; otherwise, it is considered a data packet dropped by the physical network. The first judgment criteria include: the protocol type of the data packet is VXLAN, it is encapsulated by UDP protocol and the destination port of the UDP protocol is the standard port of VXLAN; The second criterion includes: the source interface name of the data packet contains any one of the words docker0, cni0, vxlan0, flannel, and cali+.
[0012] Furthermore, the OVS-based data plane capture strategy includes: By using dynamic flow table injection technology, all flow table rules containing packet loss actions are obtained in the OVS control plane. After rewriting and transforming the action instructions, container network packets dropped by OVS are sent to a preset dedicated packet capture port to collect dropped packets from the container network.
[0013] Furthermore, the filtering process of the packet processing layer includes: filtering out the expected container network packet loss to obtain the abnormal container network dropped packets. The expected container network packet loss includes: multicast / broadcast packets not within the container network and packets that are expected to be intercepted based on network policies. When the packet processing layer forwards the packet to the container network packet loss database, it also aggregates packets with the same five-tuple information based on a preset time window. After a time window is reached, the aggregated packets are forwarded to the container network packet loss database. The packet processing layer also appends packet loss context information to each packet. This packet loss context information includes a VNI identifier, a microsecond-level timestamp of the packet loss occurrence, a physical node IP address, and a hostname. This packet loss context information is sent to the container network packet loss database along with the five-tuple information. The packet loss data analyzer, while locating the location of the dropped packet, also locates the network tenant based on the VNI identifier, thus achieving tenant location of the dropped packet; it locates the timestamp of the packet loss based on the microsecond-level timestamp; it locates the physical node where the packet loss occurred based on the physical node IP address; and it locates the host where the packet loss occurred based on the hostname.
[0014] Furthermore, the container network packet loss database is a two-layer storage architecture that separates the write layer and the read layer, and uses the buffer engine table as the write layer. The container network packet loss database uses a low cardinality string type to store fields with high repetition in data packets; and it performs partitioned storage based on the timestamp of the data packets.
[0015] Furthermore, the process by which the packet loss data analyzer locates the position where the dropped data packet occurred includes: Based on the pre-acquired container cluster topology information, obtain a container cluster map containing all microservices, service entry points and service endpoints, as well as their locations and statuses; Based on the five-tuple information corresponding to each data packet stored in the container network packet loss database, each data packet is associated with the container cluster map through the mapping relationship between IP address and port, and the specific location where the dropped data packet occurred is identified. Record the complete path of data packets from each microservice in the container cluster map, through the corresponding service entry point, to the service endpoint, and obtain the packet loss status at each stage to infer the root cause of data packet loss.
[0016] Furthermore, the rules for generating alarms by the packet loss data analyzer include: Alerts are issued based on the packet loss rate and the duration of packet loss. An alert is issued based on the upward trend of the packet loss rate; Alerts are generated based on the number of microservices or service entry points affected by dropped data packets.
[0017] The present invention also provides a packet loss monitoring method for the container network packet loss monitoring system as described above, comprising the following steps: Capture packets dropped by the container network and extract the five-tuple information from the captured packets, which includes the source IP address, destination IP address, source port, destination port, and transport protocol; Store the captured data packets and the extracted quintuple information; Based on the pre-acquired container cluster topology information, and according to the five-tuple information corresponding to each data packet stored, the microservice components associated with the data packets in the container cluster topology information are determined, the location of the dropped data packets is located, and an alarm is generated.
[0018] Compared with the prior art, the present invention has the following advantages: (1) This invention captures dropped data packets in the container network by deploying a container network packet loss collector on each node of the container cluster and extracts the five-tuple information therein without affecting the operation of the container cluster; the five-tuple information of the data packets is stored and transmitted to the packet loss data analyzer, which can effectively reduce the amount of information written to the database and greatly reduce the amount of data that needs to be uploaded; by combining the collected five-tuple information and cluster information through the packet loss data analyzer, the data packet path can be accurately restored and the packet loss location can be located, which provides great convenience for the operation and maintenance of ultra-large-scale container networks.
[0019] (2) The container network packet loss collector of the present invention adopts a layered design, including a packet capture layer and a packet processing layer. The packet capture layer deploys probes with different capture strategies for the data plane based on the Linux kernel network protocol stack and the data plane based on OVS, respectively, in order to capture the packets dropped by the container network. In the data plane capture strategy based on the Linux kernel network protocol stack, by attaching probes based on eBPF programs to the ingress and egress queues of the TC subsystem that data packets must pass through in the Linux kernel, the execution of eBPF programs can be triggered before the execution of the node's filtering rules. This is beneficial for identifying the precise time and location of data packet drop, and it runs in the Linux kernel, which is highly efficient. In the OVS-based data plane capture strategy, all flow table rules containing packet loss actions in the OVS control plane are directly rewritten and transformed, so that packets that should have been dropped are automatically sent to the dedicated packet capture port, which basically does not increase the amount of additional computation and has high running efficiency.
[0020] (3) The present invention also noted that the data packets obtained by the capture strategy based on the data plane of the Linux kernel network protocol stack include both container network and physical network packet loss. It proposes to determine whether the data packet is a container network packet loss by using the protocol characteristics of packet loss and network interface information, thereby improving the accuracy of data collection.
[0021] (4) The packet processing layer in the container network packet loss collector of the present invention first performs filtering to filter out those container network packet losses that meet the expectations, including multicast / broadcast that are not within the scope and network policy expected interception. The remaining ones are the abnormal packet losses of the container network that need to be paid attention to; it can avoid invalid alarms caused by a large number of normal packet losses and improve the efficiency of problem investigation. During the data transformation process of the data packet processing layer, packet loss information is also aggregated in the time dimension, which can effectively reduce the number of database writes.
[0022] (5) Considering the following characteristics of packet loss data in container networks: fixed data fields, all write operations without update operations, and time series characteristics, this invention proposes that the container network packet loss database use a columnar database as the core storage engine to persistently store packet loss data from the data aggregation layer of each node; and designs a two-layer storage architecture to separate the write layer and the read layer, effectively alleviating the pressure of simultaneous writing by ultra-large-scale nodes; uses a low cardinality string type to store fields with high repetition, reducing storage space occupation; and optimizes the data storage structure by partitioning the packet loss data timestamp, which facilitates querying according to time windows and controlling the data lifecycle.
[0023] (6) In the process of locating the packet loss data analyzer, the present invention associates discrete packet loss events with specific microservice components through IP address and port mapping relationship based on the obtained cluster topology information, identifies the specific location where packet loss occurs, and can realize full-link packet loss traffic analysis to obtain the packet loss rate and latency of each link, which is convenient for inferring the possible root causes of abnormal packet loss. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the framework of a container network packet loss monitoring system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the framework of a container network packet loss collector provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the processing flow of a container network packet loss collector provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the framework of a packet loss data analyzer provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating a container network packet loss monitoring method provided in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0026] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0027] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0028] In the description of this invention, some keywords are defined as follows: Containers: A lightweight virtualization technology that packages applications and their dependencies into a single, portable unit, decoupling applications from infrastructure. They feature resource isolation, rapid startup, a consistent environment, and efficient deployment, providing an ideal runtime environment for microservice architectures and cloud-native applications, and significantly improving application delivery and operational efficiency.
[0029] Kubernetes is an open-source container orchestration platform for automating the deployment, scaling, and management of containerized applications. It provides core functionalities such as declarative configuration, automated deployment, service discovery, load balancing, and self-healing capabilities, effectively solving resource scheduling, application management, and service coordination issues in large-scale distributed systems. It has become the de facto standard for cloud-native infrastructure.
[0030] Container networking is a virtual network layer built on top of existing physical networks. It creates logical connections through software-defined methods, enabling network nodes to communicate across physical network boundaries. It is widely used in scenarios such as container clusters and cloud computing, and has good isolation and flexibility.
[0031] Example 1 like Figure 1 As shown, this embodiment provides a container network packet loss monitoring system, including: The container network packet loss collector is deployed on each node of the container cluster. The container network is built on the container cluster and is used to capture packets dropped by the container network and extract the five-tuple information from the captured packets. The five-tuple information includes the source IP address, destination IP address, source port, destination port and transport protocol. The container network packet loss database stores the five-tuple information of data packets captured by the container network packet loss collector. The packet loss data analyzer is used to determine the microservice components associated with the packets in the container cluster topology information based on the pre-acquired container cluster topology information and the five-tuple information corresponding to each packet stored in the container network packet loss database, locate the location of the dropped packets, and generate alarms.
[0032] The composition and function of each part are described in detail below.
[0033] 1. Container network packet loss collector The container network packet loss collector is deployed on each node of the container cluster. It can capture actively dropped packets in the container network in real time, extract the five-tuple feature information (source IP, destination IP, source port, destination port, and transport protocol), and integrate this information to store it in a remote database. This provides critical data support for container network troubleshooting and performance optimization. The container network packet loss collector designed in this solution is compatible with both the Linux kernel's native data plane and the OpenvSwitch (OVS) data plane, and can be adapted to various container network data plane environments.
[0034] Figure 2 This section demonstrates the structure of a container network packet loss collector. The collector employs a layered design, primarily consisting of a packet capture layer and a packet processing layer. The functions of each layer will be described below.
[0035] 1.1 Packet Capture Layer Most existing container network data plane solutions can be divided into two categories: data planes based on the Linux kernel network protocol stack and data planes based on OVS. These two types of data planes have different packet flow paths, so different capture techniques are needed for each data plane path to detect abnormal packet loss in container networks.
[0036] The packet capture layer is used to identify the data plane of packets dropped by the container network. Different capture strategies are used to deploy probes for the data plane based on the Linux kernel network protocol stack and the data plane based on OVS, in order to capture packets dropped by the container network. Data plane capture strategies based on the Linux kernel network protocol stack include: Based on eBPF technology, packet loss capture probes are deployed on the ingress and egress queues of critical network nodes in the Linux kernel, and the classifiers of critical network nodes are modified so that the packet loss capture logic is triggered before the filtering rules of the critical network nodes are executed. For packets captured by the packet loss capture probe, the protocol characteristics and network interface information of the packets are used to determine whether they are packets dropped by the container network, and then they are sent to the packet processing layer.
[0037] That is, for packet loss in the native data plane of the Linux kernel, the packet capture layer uses the eBPF (extended Berkeley Packet Filter) program to attach to the ingress and egress queues of the TC (Traffic Control) subsystem. By modifying the TC classifier, the capture logic is triggered before the packet matches the filtering rule with the drop action.
[0038] The TC subsystem is the kernel's flow control module, acting as a tollbooth or sorting center with rules and filters that data packets must pass through. It can queue, shape, and filter data packets.
[0039] The TC classifier is a rulebook or sorting rule within the TC subsystem. When a data packet arrives, it is matched against these rules to determine whether it should be forwarded, modified, or dropped.
[0040] An eBPF program is a tiny program that can be safely injected into the kernel to run.
[0041] Note that the captured Linux kernel data packets include both container network and physical network packet loss. Therefore, it is necessary to determine whether the data packet is a container network packet loss by using the protocol characteristics of the packet loss and the network interface information.
[0042] The process of determining whether a data packet is dropped by the container network includes: determining whether the data packet meets either the first or the second preset judgment criteria. If it does, it is considered a data packet dropped by the container network; otherwise, it is considered a data packet dropped by the physical network. The first criterion includes: the protocol type of the data packet is VXLAN (Virtual eXtensible Local Area Network), it is encapsulated by UDP protocol and the destination port of the UDP protocol is the standard port of VXLAN, namely port 4789; The second criterion includes: the source interface name of the data packet contains any one of the words docker0, cni0, vxlan0, flannel, and cali+, which are characteristics of typical container network interface names.
[0043] If either of the above two conditions is met, it is considered a container network packet loss. Container network packet loss that meets the conditions will be sent to the data processing layer for further processing.
[0044] OVS-based data plane capture strategies include: By using dynamic flow table injection technology, all flow table rules containing packet loss actions are obtained in the OVS control plane. After rewriting and transforming the action instructions, container network packets dropped by OVS are sent to a preset dedicated packet capture port to collect dropped packets from the container network.
[0045] Here is an example of a conversion: Original flow table rules: table=0,priority=100,ip,nw_src=10.0.0.1 actions=drop Rewritten flow table rules: table=0,priority=100,ip,nw_src=10.0.0.1 actions=output: CAPTURE_PORT, drop.
[0046] Through flow table rewriting, container network packets that would otherwise be discarded by OVS are sent to a dedicated packet capture port. This port does not participate in normal network data forwarding; it only accepts container network packets that need to be analyzed. Subsequent data processing layers will receive these packets and perform further processing.
[0047] 1.2 Data Packet Processing Layer The packet processing layer is used to filter and extract the 5-tuple information from the packets acquired by the packet capture layer, and then forward them to the container network packet loss database.
[0048] The filtering process includes: filtering out expected container network packet loss, obtaining abnormal container network dropped packets, and identifying abnormal container network packet loss that needs attention. Expected container network packet loss includes, but is not limited to: multicast / broadcast packets outside the container network, and packets that are expected to be blocked based on network policies, etc.
[0049] Out-of-scope multicast / broadcast: Container networks are typically overlay networks, and broadcast domains are carefully designed to be very small. A broadcast / multicast packet destined for the entire network or a specific subnet should be expected to be dropped by the local network layer if the destination is not within a container on the current host; this is normal network isolation behavior. Network policy expected interception: If the administrator sets a network policy for the order service that prohibits the front-end service from accessing its management port, then all such access attempts should be dropped. This packet loss is a successful execution of the security policy.
[0050] After filtering, the packet processing layer extracts the five-tuple information of abnormal packet loss in the container network, including source IP address, destination IP address, source port, destination port, and protocol type.
[0051] At the same time, the processing layer will add packet loss context information, including the packet VNI (VXLAN Network Identifier), the microsecond-level timestamp of the packet loss, the physical node IP address, and the hostname.
[0052] VNI is used to locate network tenants, enabling tenant identification of packet loss. Packet loss timestamp is used to locate the timestamp when packet loss occurred. Physical node IP is used to locate the physical node where packet loss occurred.
[0053] The processed abnormal packet loss information needs to be written to a remote database for persistent storage.
[0054] To reduce the number of database writes, the packet processing layer also needs to aggregate local abnormal packet loss information over time, with an aggregation time window typically ranging from 10 to 30 seconds. Within a time window, packet loss information with identical 5-tuples is aggregated together, and the number of packet losses within the window is counted. After the aggregation period ends, the packet processing layer writes the aggregated abnormal packet loss information to the remote database.
[0055] 1.3 Workflow of the Container Network Packet Loss Collector Figure 3 The workflow of the container network packet loss collector is demonstrated, including the following five steps: Step 1.1: Container Network Data Plane Identification. In this stage, the system first accurately identifies packet loss in the container network environment. This includes monitoring packet loss in both the Linux kernel native data plane and the OVS data plane. The system will distinguish between different types of packet loss events, paying particular attention to packet loss in the overlay network, in preparation for subsequent processing.
[0056] Step 1.2: Deploy capture probes. Based on the identification results from the previous step, the system deploys dedicated packet loss capture probes at key network nodes. For container network data planes based on the Linux kernel protocol stack, the probes are primarily implemented using eBPF technology, enabling them to be attached to critical kernel event points. For the OVS environment, probes are deployed through a flow table extension mechanism. The probe deployment adopts a lightweight design to minimize the performance impact on the production environment while maintaining the integrity of the capture capabilities.
[0057] Step 1.3: Enable packet loss capture in the container network. This stage activates the deployed probes and begins real-time capture of packet loss events in the container network. No data packets are parsed during the capture process to ensure efficient data collection.
[0058] Step 1.4: The packet processing layer processes and aggregates lost packet data. This stage processes captured packets, including expected packet loss filtering, extraction of packet 5-tuple information, context information enhancement, and data aggregation. This step effectively reduces the amount of information written to the database.
[0059] Step 1.5: Send lost data to the database. In this stage, the collector sends the aggregated data to the remote database for persistent storage.
[0060] 2. Container network packet loss database Packet loss data in container networks has the following characteristics: fixed data fields, all write operations without update operations, and time-series characteristics. Based on these characteristics, this patent considers using a columnar database as the core storage engine to persistently store packet loss data from the data aggregation layer of various nodes. Common columnar databases, such as ClickHouse and Apache Druid, feature high data compression ratios, high performance for time-series data queries, and distributed architectures that support horizontal scaling, making them well-suited for the scenario of storing packet loss data in ultra-large-scale container networks.
[0061] Preferably, to adapt to packet loss scenarios in ultra-large-scale container networks, this solution makes the following optimizations to the database: 2.1 Dual-layer storage architecture design: To alleviate the pressure of simultaneous writes from ultra-large-scale nodes, the write layer and read layer are separated. A buffer engine table is used as the front-end write layer to effectively cope with sudden storm scenarios. It batch buffers potentially hundreds of thousands of packet loss records per second in memory and automatically inserts them in batches, reducing the pressure on the back-end storage engine. The subsequent read layer ensures the consistency and reliability of data reading through automatic replica synchronization and deduplication functions.
[0062] 2.2 Data Structure Storage Optimization: Fields with high repetition rates, such as node names and IP addresses, are stored using low-cardinality string types to reduce storage space usage. Data is partitioned using timestamps of lost packets, facilitating querying by time window and controlling data lifecycle.
[0063] 3. Packet Loss Data Analyzer like Figure 4 As shown, the container network packet loss analyzer mainly has two functions: packet loss location and alarm generation. The container network packet loss analyzer is responsible for performing in-depth analysis and processing of the collected packet loss data, locating the packet loss location, and generating alarms for the detected packet loss problems to send to network operation and maintenance personnel.
[0064] 3.1 Packet Loss Location The process by which a packet loss data analyzer locates the position of dropped data packets includes: Based on the pre-acquired container cluster topology information, obtain a container cluster map containing all microservices, service entry points and service endpoints, as well as their locations and statuses; Based on the five-tuple information corresponding to each data packet stored in the container network packet loss database, each data packet is associated with the container cluster map through the mapping relationship between IP address and port, and the specific location where the dropped data packet occurred is identified. Record the complete path of data packets from each microservice in the container cluster map, through the corresponding service entry point, to the service endpoint, and obtain the packet loss status at each stage to infer the root cause of data packet loss.
[0065] Specifically, packet loss localization is the core functional module of this system. Through multi-dimensional data correlation and intelligent analysis, it achieves accurate localization of network faults in Kubernetes clusters. This module first queries packet loss data within a specific time window from the time-series database, and then combines it with the existing physical network topology to construct a complete data packet transmission path map.
[0066] During the correlation analysis phase, the system fully utilizes cluster topology information obtained from the Kubernetes API, including detailed metadata of resources such as microservices (Pods), service entry points (Services), and service endpoints (Endpoints). Through IP address and port mapping relationships, discrete packet loss events are associated with specific microservice components.
[0067] In particular, the system can accurately identify the specific location where packet loss occurs: it could be the egress network card of the source Pod, the ingress network card of the target Pod, the forwarding link of the intermediate node, or the Service load balancing layer.
[0068] For Service resources, the analyzer implements end-to-end traffic analysis. By recording the complete path of data packets from the source Pod through the Service load balancer and Endpoint forwarding to the target Pod, it obtains the packet loss rate and latency at each stage. When abnormal packet loss is detected, the analyzer can automatically infer possible root causes, such as excessive node load, network policy conflicts, Service configuration errors, or abnormal Endpoint health status.
[0069] 3.2 Alarm Generation The alarm generation function is responsible for promptly notifying relevant personnel of detected network packet loss issues, enabling early detection and resolution of problems. This function supports highly flexible alarm rule configuration, ensuring that critical network issues quickly reach the appropriate personnel. The system provides multi-dimensional alarm rule capabilities as follows: Threshold-based: Set thresholds such as packet loss rate and duration to trigger alarms; Trend-based: Detecting trend changes such as a rapid increase in packet loss rate; Based on the scope of impact: Adjust the severity level according to the number of Pods / Services affected; Based on business importance: Allows setting more sensitive alerting policies for critical business operations.
[0070] To avoid alarm storms and alarm fatigue, the alarm generation function also implements the following intelligent alarm processing: Alarm deduplication: Identify and merge multiple alarms originating from the same root cause; Alarm suppression: When an upstream component failure is detected, related alarms from downstream components are automatically suppressed; Alarm silencing: Supports temporarily silencing specific types of alarms during maintenance windows.
[0071] like Figure 5 As shown, this embodiment also provides a packet loss monitoring method for the container network packet loss monitoring system described above, including the following steps: S1: Capture packets dropped by the container network and extract the five-tuple information from the captured packets. The five-tuple information includes the source IP address, destination IP address, source port, destination port, and transport protocol. S2: Store the captured data packets and extracted quintuple information; S3: Based on the pre-acquired container cluster topology information, and according to the five-tuple information corresponding to each data packet stored, determine the microservice component associated with the data packet in the container cluster topology information, locate the location where the dropped data packet occurred, and generate an alarm.
[0072] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A container network packet loss monitoring system, characterized in that, include: A container network packet loss collector is deployed on each node of a container cluster, which is built on the container cluster. It is used to capture packets dropped by the container network and extract the five-tuple information from the captured packets. The five-tuple information includes the source IP address, destination IP address, source port, destination port and transport protocol. The container network packet loss database stores the five-tuple information of data packets captured by the container network packet loss collector. The packet loss data analyzer is used to determine the microservice components associated with the packets in the container cluster topology information based on the pre-acquired container cluster topology information and the five-tuple information corresponding to each packet stored in the container network packet loss database, locate the location of the dropped packets, and generate alarms.
2. The container network packet loss monitoring system according to claim 1, characterized in that, The container network packet loss collector includes a packet capture layer and a packet processing layer; The packet capture layer is used to identify the data plane of packets dropped by the container network. Different capture strategies are used to deploy probes for the data plane based on the Linux kernel network protocol stack and the data plane based on OVS, respectively, in order to capture packets dropped by the container network. The packet processing layer is used to filter and extract the five-tuple information of the packets obtained by the packet capture layer, and then forward them to the container network packet loss database.
3. The container network packet loss monitoring system according to claim 2, characterized in that, The data plane capture strategy based on the Linux kernel network protocol stack includes: Based on eBPF technology, packet loss capture probes are deployed on the ingress and egress queues of critical network nodes in the Linux kernel, and the classifiers of the critical network nodes are modified so that the packet loss capture probes trigger the capture logic before the packets are executed by the filtering rules of the critical network nodes. For the data packets captured by the packet loss capture probe, it is determined whether the data packets are dropped by the container network based on the protocol characteristics and network interface information of the data packets, so as to send them to the data packet processing layer.
4. The container network packet loss monitoring system according to claim 3, characterized in that, The process of determining whether a data packet is dropped by the container network includes: determining whether the data packet meets either a preset first judgment criterion or a preset second judgment criterion; if it meets the criteria, it is considered a data packet dropped by the container network; otherwise, it is considered a data packet dropped by the physical network. The first judgment criteria include: the protocol type of the data packet is VXLAN, it is encapsulated by UDP protocol and the destination port of the UDP protocol is the standard port of VXLAN; The second criterion includes: the source interface name of the data packet contains any one of the words docker0, cni0, vxlan0, flannel, and cali+.
5. A container network packet loss monitoring system according to claim 2, characterized in that, The OVS-based data plane capture strategy includes: By using dynamic flow table injection technology, all flow table rules containing packet loss actions are obtained in the OVS control plane. After rewriting and transforming the action instructions, container network packets dropped by OVS are sent to a preset dedicated packet capture port to collect dropped packets from the container network.
6. A container network packet loss monitoring system according to claim 2, characterized in that, The filtering process of the packet processing layer includes: filtering out the expected container network packet loss to obtain the abnormal container network dropped packets. The expected container network packet loss includes: multicast / broadcast packets outside the container network and packets that are expected to be blocked based on network policies. When the packet processing layer forwards the packet to the container network packet loss database, it also aggregates packets with the same five-tuple information based on a preset time window. After a time window is reached, the aggregated packets are forwarded to the container network packet loss database. The packet processing layer also appends packet loss context information to each packet. This packet loss context information includes a VNI identifier, a microsecond-level timestamp of the packet loss occurrence, a physical node IP address, and a hostname. This packet loss context information is sent to the container network packet loss database along with the five-tuple information. The packet loss data analyzer, while locating the location of the dropped packet, also locates the network tenant based on the VNI identifier, thus achieving tenant location of the dropped packet; it locates the timestamp of the packet loss based on the microsecond-level timestamp; it locates the physical node where the packet loss occurred based on the physical node IP address; and it locates the host where the packet loss occurred based on the hostname.
7. A container network packet loss monitoring system according to claim 1, characterized in that, The container network packet loss database is a two-layer storage architecture that separates the write layer and the read layer, and uses the buffer engine table as the write layer. The container network packet loss database uses a low cardinality string type to store fields with high repetition in data packets; and it performs partitioned storage based on the timestamp of the data packets.
8. The container network packet loss monitoring system according to claim 1, characterized in that, The process by which the packet loss data analyzer locates the position of the dropped data packets includes: Based on the pre-acquired container cluster topology information, obtain a container cluster map containing all microservices, service entry points and service endpoints, as well as their locations and statuses; Based on the five-tuple information corresponding to each data packet stored in the container network packet loss database, each data packet is associated with the container cluster map through the mapping relationship between IP address and port, and the specific location where the dropped data packet occurred is identified. Record the complete path of data packets from each microservice in the container cluster map, through the corresponding service entry point, to the service endpoint, and obtain the packet loss status at each stage to infer the root cause of data packet loss.
9. A container network packet loss monitoring system according to claim 1, characterized in that, The rules for generating alarms by the packet loss data analyzer include: Alerts are issued based on the packet loss rate and the duration of packet loss. An alert is issued based on the upward trend of the packet loss rate; Alerts are generated based on the number of microservices or service entry points affected by dropped data packets.
10. A packet loss monitoring method for a container network packet loss monitoring system as described in any one of claims 1-9, characterized in that, Includes the following steps: Capture packets dropped by the container network and extract the five-tuple information from the captured packets, which includes the source IP address, destination IP address, source port, destination port, and transport protocol; Store the captured data packets and the extracted quintuple information; Based on the pre-acquired container cluster topology information, and according to the five-tuple information corresponding to each data packet stored, the microservice components associated with the data packets in the container cluster topology information are determined, the location of the dropped data packets is located, and an alarm is generated.