An intelligent analysis and repair system for network link exception under a domestic operating system

Abstract

This invention relates to an intelligent analysis and repair system for network link anomalies under a domestically developed operating system, belonging to the field of network communication. The system comprises a collection layer, a feature layer, an analysis layer, a decision-making and repair layer, a presentation layer, and a knowledge layer. The collection layer collects network performance and packet loss metrics from multiple dimensions, including the physical network interface card (NIC) layer, ring buffer layer, soft interrupt and NAPI layer, netdev_max_backlog layer, protocol stack layer, and socket buffer layer. The feature layer transforms multi-source heterogeneous data into analytical features and establishes cross-layer correlations. The analysis layer uses an engine combining a preset rule base and a preset machine learning model to identify packet loss scenarios and locate root causes. The decision-making and repair layer generates and executes repair suggestions based on a policy knowledge base, supporting step-by-step adjustments and automatic rollback. The presentation layer provides a visual operation and maintenance interface. The knowledge layer accumulates and processes case studies and iteratively optimizes the model. This invention achieves a closed-loop intelligent analysis and automated repair of packet loss across the entire link from the physical NIC to the application process, improving the network operation and maintenance efficiency of domestically developed operating systems.

Description

Technical Field

[0001] This invention relates to the field of network communication, and in particular to an intelligent analysis and repair system for network link anomalies under a domestically developed operating system. Background Technology

[0002] With the deepening of the information technology innovation project in key industries such as finance, power, and telecommunications, a large number of servers have completed the migration of domestic operating systems. Network systems need to meet the core requirements of high reliability, high throughput, and low latency. Network packet loss will directly affect the stability of critical services and user experience.

[0003] Existing methods for troubleshooting network packet loss suffer from several drawbacks, including fragmented investigation dimensions, reliance on manual experience for pinpointing the source, lack of automated repair loops, and insufficient compatibility with domestic operating systems. Furthermore, traditional statistical methods based on ` / proc / sys` have limited granularity, failing to accurately trace the specific code path and process instance of packet loss. While eBPF technology enables low-overhead, fine-grained observation of the Linux kernel, current solutions primarily focus on general monitoring, lacking a comprehensive solution for intelligent analysis and automatic repair of the entire network packet loss chain, tailored to domestic operating systems. Summary of the Invention

[0004] To achieve accurate location and automated repair of packet loss root causes, this invention provides an intelligent analysis and repair system for network link anomalies under a domestically developed operating system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A network link anomaly intelligent analysis and repair system under a domestic operating system includes a collection layer, a feature layer, an analysis layer, a decision repair layer, a display layer, and a knowledge layer. The collection layer, feature layer, analysis layer, and decision repair layer are sequentially and communicatively connected. The collection layer, feature layer, analysis layer, and decision repair layer are all communicatively connected to the display layer. The knowledge layer is communicatively connected to both the display layer and the analysis layer. The acquisition layer is used to collect multi-source heterogeneous data on network operation status from multiple dimensions. This includes periodically collecting network performance and packet loss indicators of the physical network card layer, network card ring buffer layer, soft interrupt and NAPI layer, netdev_max_backlog layer, protocol stack layer and socket buffer layer by reading / proc, sys file system interface and system command calls. It also uses a dual mechanism based on eBPF technology, which prioritizes static tracepoint adaptation and uses dynamic function symbol resolution as a fallback. Customized eBPF programs are mounted at key functions and tracepoints of the network subsystem of the domestic operating system kernel to capture the flow of data packets in each layer of the protocol stack, abnormal packet loss events and socket layer behavior data. The feature layer is used to perform time alignment and correlation on the multi-source heterogeneous data acquired by the acquisition layer, transforming it into features that can be used for analysis and modeling, and establishing cross-level correlation relationships. The analysis layer is used to automatically identify typical packet loss scenarios and locate the root cause of packet loss based on the features and relationships generated by the feature layer and an analysis engine that combines a preset rule base and a preset machine learning model. The decision repair layer is used to generate and execute optimal or suboptimal repair suggestions based on the built-in strategy knowledge base specifically designed for tuning domestic operating systems after the analysis layer determines the root cause of packet loss. The presentation layer is used to provide a unified visual operation and maintenance interface, which displays packet loss detection results, root cause location conclusions, repair execution process and verification results in the form of link topology diagram, key indicator trend diagram and event timeline. It supports multi-dimensional filtering and data drilling by host, network card interface, service tag, process name and network connection. The knowledge layer is used to automatically transform each successful or failed packet loss handling case into structured knowledge and deposit it into a centralized knowledge base. The newly deposited cases are used periodically to perform incremental training and iterative optimization on the preset rule base and preset machine learning model.

[0006] Optionally, the feature layer establishes cross-level associations including: The raw metrics are aggregated using a fixed time window to generate a continuous sequence of metrics with precise timestamps for each network interface, ring buffer queue, network connection, and application process. By utilizing system configuration information and kernel object relationships, a full-link topology mapping is dynamically established from "physical network card → ring buffer → trigger soft interrupt handling task → netdev_max_backlog queue → protocol layer queue → socket buffer → application process"; Associating eBPF-captured function-level events with the end-to-end topology mapping establishes a precise correspondence between "code execution path → queue / interface → process / connection".

[0007] Optionally, the analysis layer uses a preset rule base to identify packet loss scenarios, including: Physical layer packet loss determination: If the packet loss count of the physical network card increases and is accompanied by link protocol errors, it is determined to be a link quality problem; if the network card speed is inconsistent with that of the switch, it is determined to be a duplex / speed mismatch. Ring buffer overflow determination: If the receive / transmit queue is full and the overflow count continues to increase, it is determined that the hardware queue overflow is caused by a sudden traffic surge. Soft interrupt packet loss determination: If the packet loss count of a certain CPU increases during the soft interrupt phase and the NAPI polling time exceeds the threshold, it is determined to be a CPU processing bottleneck. Backlog queue packet loss determination: If the netdev_max_backlog queue length continuously reaches the kernel parameter limit and branches dropped due to the queue being full are captured, it is determined to be protocol stack entry congestion; Protocol stack packet loss determination: If the TCP retransmission rate increases abnormally or packet loss is observed due to the receive window being zero, it is determined to be a protocol layer flow control or memory pressure issue. Socket buffer packet loss determination: If the socket receive buffer usage rate is consistently 100% and the application has not called recvmsg for a long time, it is determined that the application is reading slowly, causing the buffer to overflow.

[0008] Optionally, the decision-making repair layer generates repair suggestions including: Dynamic adjustment of kernel parameters: adjust the ring buffer size, set the CPU affinity of soft interrupts, modify interrupt merging parameters, increase net.core.netdev_max_backlog or socket buffer size; CPU and network optimization: Bind business processes to specific CPU cores based on their affinity, configure RPS / RFS to balance multi-core load, and adjust the number of network interface card queues; Application-side optimization suggestions: To address the process bottleneck identified by eBPF, we propose adding a packet receiving thread, optimizing data processing logic, and introducing an asynchronous I / O model; Combinatorial optimization scheme: For complex packet loss scenarios with multiple coupled factors, a combination optimization strategy is generated and executed sequentially, and a safety boundary is set for the adjustment action of each parameter.

[0009] Optionally, the decision-making repair layer may implement repair recommendations including: For network server nodes that are included in the operation and maintenance authorization scope and are within the preset whitelist, and are not during peak hours of critical business, parameter modification commands are automatically issued and executed through sysctl, ethtool, driver parameter files, and application configuration files. A step-by-step adjustment strategy is adopted for key kernel parameters. Each adjustment only increases the parameter value by a preset proportion. After the adjustment, the observation period begins. If the key indicators improve and there are no new anomalies, the next adjustment will continue. If the indicators deteriorate or an alarm is triggered, the system will immediately and automatically roll back to the previous stable state.

[0010] Optional, also includes: After the repair action is completed, the decision repair layer continuously monitors various indicators of the target link or queue, compares and evaluates the current key performance indicators with the baseline data before the repair, and if the improvement effect reaches the preset threshold, the adjusted parameters will be fixed; if the target is not met, a new round of root cause analysis will be automatically triggered.

[0011] Optional, also includes: The decision-making and repair layer meticulously records the parameter changes, execution time, and before-and-after effect comparison data for each repair action, forming a complete operation audit log that provides raw materials for subsequent case analysis and knowledge accumulation.

[0012] Optionally, the observation period may last 3-5 minutes.

[0013] Optionally, the system supports a standalone deployment mode.

[0014] In summary, the present invention has at least one of the following beneficial technical effects: 1. It pioneered a customized eBPF observation architecture for domestic operating systems, breaking through the limitations of general eBPF solutions that are strongly coupled to kernel versions and have poor compatibility with custom drivers. Through a dual mechanism of static tracepoint priority adaptation and dynamic function symbol resolution as a fallback, it ensures the stable mounting of 200+ key probes in the Kylin V10 kernel and achieves function-level attribution for 7 types of packet loss causes, including hardware packet loss, ring buffer overflow, soft interrupt latency, netdev_max_backlog discarding, protocol stack active discarding, socket buffer overflow, and application layer processing bottlenecks.

[0015] 2. Construct the first three-dimensional correlation feature model of time-topology-code across seven network layers (physical layer ring buffer interrupt backlog protocol stack socket application), strictly aligning discrete traditional statistical indicators with eBPF events on the time axis (30s sliding window), spatial axis (interface / queue / process / connection), and execution axis (function call stack / branch condition), supporting accurate link tracing.

[0016] 3. A hybrid intelligent analysis paradigm of "rule-driven as the main approach and model enhancement as the supplementary approach" is proposed. The rule base is deeply embedded with the kernel behavior characteristics unique to domestic operating systems (such as the small default value of netdev_max_backlog in Euler kernel and the difference in ring buffer reset logic in Kylin driver). The machine learning model is preset to be activated only in the rule coverage blind area, taking into account accuracy, interpretability and generalization ability in the domestic environment.

[0017] 4. An automated repair engine with a security sandbox and effect verification is designed. The policy knowledge base has built-in security boundary constraints for parameters of domestic platforms (such as netdev_max_backlog adjustment not exceeding twice the current value and not exceeding the memory limit), CPU binding compliance checks, RPS queue number and physical core number matching verification mechanisms, and enforces the automatic rollback loop process of hot update of snapshot before adjustment for 5 minutes steady state observation KPI achievement judgment failure, which completely solves the core pain point of traditional solutions "adjustment does not work, and there is no way to roll back if it is wrong". Attached Figure Description

[0018] Figure 1 This is a system framework diagram of a network link anomaly intelligent analysis and repair system under a domestic operating system according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of a network link anomaly intelligent analysis and repair system under a domestic operating system, according to an embodiment of this application.

[0020] Figure 3 This is a user flowchart of a network link anomaly intelligent analysis and repair system under a domestic operating system, according to an embodiment of this application. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of the present invention.

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Existing methods for investigating network packet loss face the following core challenges: The investigation dimensions are scattered: indicators from various layers such as physical network cards, ring buffers, soft interrupts, protocol stacks, and socket buffers are scattered across different interfaces, making it impossible to conduct unified correlation analysis; Location is highly dependent on human experience: it requires maintenance personnel to be proficient in the working mechanism of the Linux kernel network subsystem, resulting in low troubleshooting efficiency and a high operational threshold. Lack of automated repair loop: After locating the root cause, kernel parameters need to be manually modified, which can easily introduce new risks due to improper parameter settings, and there is a lack of a mechanism to verify the repair effect; Insufficient compatibility with domestic operating systems: Existing tools are designed for general Linux distributions and lack sufficient support for kernel versions, parameter differences, and driver customization of domestic operating systems. Incomplete indicator collection and inaccurate analysis rules also contribute to the problem. Limited observation granularity: Traditional statistical methods based on / proc / sys provide aggregated results, which cannot accurately trace the specific code path, process instance, and network flow of packet loss. Although eBPF technology can achieve low-overhead, fine-grained observation of the Linux kernel, existing solutions mostly focus on general monitoring scenarios and have not yet formed an integrated solution for domestic operating systems that aims at intelligent analysis and automatic repair of the entire network packet loss chain.

[0025] This invention provides a network packet loss intelligent analysis and repair system based on a domestic operating system. It integrates traditional kernel indicator collection with eBPF fine-grained observation technology, and opens up the entire link observation from physical network card → ring buffer → triggering soft interrupt handling task → netdev_max_backlog → TCP / IP protocol stack → socket buffer → application process, and constructs cross-layer time sequence features and topology correlation. This solution is significantly superior to existing solutions from competitors in terms of technical effectiveness. The root cause of packet loss can be located precisely in each network link path, which is two orders of magnitude higher than traditional tools. The time required for end-to-end troubleshooting has been reduced from hours to minutes (average <10 minutes), and front-line operations and maintenance personnel can complete closed-loop handling without kernel development experience; The repair success rate exceeds 90%, and the packet loss rate decreases by an average of more than 80% after repair. Furthermore, through security verification and effect tracking mechanisms, secondary failures caused by parameter misadjustment are prevented. Fully compatible with domestic operating systems, with a kernel indicator collection completeness rate of over 90%, and the eBPF probe achieving 100% stable operation rate in the kernel; For the first time, a complete autonomous closed loop has been achieved, from "observable" to "diagnosable" and then to "repairable and verifiable," which promotes the network quality assurance system of information technology innovation infrastructure from passive response to proactive immunity.

[0026] The following is an explanation of the technical terms involved in this invention: eBPF (extended Berkeley Packet Filter): An extended Berkeley packet filter that enables fine-grained kernel observation without modifying the kernel source code.

[0027] Ring buffer: A fixed-size, cyclically used lock-free data structure in the kernel, mainly used for efficient data transfer between producers and consumers, avoiding frequent memory allocation and deallocation.

[0028] Soft interrupts: The second half of the interrupt handling mechanism in the Linux kernel, used to delay the execution of time-consuming but interruptible processing tasks outside the context of hard interrupts.

[0029] netdev_max_backlog: A parameter for controlling the depth of the network protocol stack receive path queue.

[0030] Socket buffer: The receive / send buffer of the application layer socket, used for data packet interaction between the application process and the kernel protocol stack.

[0031] Tracepoint: A predefined static probe in the Linux kernel, existing in hard-coded form in the kernel source code, used to collect debugging and performance information in critical code paths (such as system call entry / exit points, network transmission and reception, scheduling events).

[0032] like Figure 1 As shown, the technical framework of this invention includes a data acquisition layer, a feature layer, an analysis layer, a decision repair layer, a presentation layer, and a knowledge layer. The data acquisition layer, feature layer, analysis layer, and decision repair layer are sequentially and communicatively connected. The data acquisition layer, feature layer, analysis layer, and decision repair layer are all communicatively connected to the presentation layer. The knowledge layer is communicatively connected to both the presentation layer and the analysis layer. Specifically: The acquisition layer is used to collect multi-source heterogeneous data on network operation status from multiple dimensions. This includes periodically collecting network performance and packet loss indicators of the physical network card layer, network card ring buffer layer, soft interrupt and NAPI layer, netdev_max_backlog layer, protocol stack layer and socket buffer layer by reading / proc, sys file system interface and system command calls (such as ifcong, netstat). Based on eBPF technology, a dual mechanism of static Tracepoint priority adaptation and dynamic function symbol resolution is adopted to attach customized eBPF programs at key functions and Tracepoints of the network subsystem of the domestic operating system kernel to capture the flow process of data packets in each layer of the protocol stack, abnormal packet loss events and socket layer behavior data. The feature layer is used to time-align and correlate the multi-source heterogeneous data acquired by the acquisition layer, transforming it into features that can be used for analysis and modeling, and establishing cross-level correlations. The analysis layer is used to identify typical packet loss scenarios and locate the root cause of packet loss based on the features and relationships generated by the feature layer. It employs an analysis engine that combines a pre-set rule base and a pre-set machine learning model. The decision repair layer is used to generate and execute optimal or suboptimal repair suggestions based on the built-in strategy knowledge base specifically designed for tuning domestic operating systems after the root cause of packet loss is determined by the analysis layer. The presentation layer provides a unified visual operation and maintenance interface, displaying packet loss detection results, root cause localization conclusions, repair execution process and verification effects in the form of link topology diagrams, key indicator trend charts and event timelines. It supports multi-dimensional filtering and data drilling by host, network card interface, service tag, process name and network connection. The knowledge layer is used to automatically transform each successful or failed packet loss handling case into structured knowledge and deposit it into a centralized knowledge base. The newly deposited cases are used periodically to perform incremental training and iterative optimization on the preset rule base and preset machine learning model.

[0033] like Figure 1 and Figure 2 As shown Example 1: Multi-level index acquisition and fine-grained observation of eBPF A comprehensive and fine-grained network operation status data acquisition layer is constructed. The acquisition layer consists of two parts: multi-level index acquisition and eBPF fine-grained observation, which together provide a data foundation for the analysis of other layers.

[0034] The specific data collection for the multi-level indicators is as follows: Deploy a kernel-level monitoring agent on the target system. By reading file system interfaces such as / proc and sys, and calling system commands (such as ifconfig and netstat), periodically collect network performance and packet loss metrics from the following six key levels. The collection frequency can be configured according to business needs, with a default value of 10 seconds per collection. Details of the metric collection are as follows: Physical network interface card (NIC) layer: including but not limited to the total number of data packets received / sent, the hardware packet loss count due to link errors or checksum failures, the number of driver error reports, and the link speed and duplex mode.

[0035] Network interface card (NIC) ring buffer layer: Monitors the depth of the receive / transmit ring queue, current usage, number of overflows due to queue fullness, and number of queue resets.

[0036] Soft interrupts and NAPI layer: Parse / proc / net / softnet_stat to obtain packet loss counts and CPU processing queue length during the soft interrupt phase, and evaluate interrupt handling latency in conjunction with kernel scheduling information.

[0037] The netdev_max_backlog layer collects the real-time length of the protocol stack ingress backup queue and packet enqueue / drop statistics, and compares them with the current upper limit of the kernel parameter net.core.netdev_max_backlog.

[0038] Protocol stack layer: For IP, TCP, and UDP protocols, collect the number of input / output packets, the number of TCP retransmissions and out-of-order packets, the length of the protocol layer receive / send queue, and the drop count caused by insufficient memory or verification failure.

[0039] Socket buffer layer: Associated with specific application processes, it collects the configured size, current usage rate, overflow and drop statistics due to buffer fullness of their socket receive / send buffers, and monitors the system call behavior of the application in blocking / non-blocking modes.

[0040] In the acquisition layer, this invention employs a dual-track observation system, consisting of a traditional network observation subsystem and an eBPF fine-grained observation subsystem operating in parallel. Traditional network observation subsystems are based on native operating system commands and packet capture tools, mainly including ifconfig, ethtool, netstat, tcpdump, and Wireshark, used to obtain interface statistics, link / driver parameters, protocol stack statistics, packet-level data, and manual visualization verification results; The eBPF fine-grained observation subsystem collects information on function-level events, call paths, abnormal branches, and process associations by attaching probes to critical kernel functions and tracepoints, enabling precise tracing of packet loss locations and code paths.

[0041] To address the shortcomings of traditional metrics in tracing the root causes of packet loss, this invention introduces eBPF technology to achieve non-invasive, low-overhead deep observation. The specific details of eBPF fine-grained observation are as follows: Based on eBPF technology, a dual mechanism of static tracepoint-first adaptation and dynamic function symbol resolution as a fallback is adopted. Customized eBPF programs are mounted at key functions and tracepoints in the network subsystem of the domestic operating system kernel to capture the flow of data packets at each layer of the protocol stack, abnormal packet loss events, and socket layer behavior data, specifically including: Critical path probe mounting: Customized eBPF programs are mounted at critical functions and tracepoints in the kernel network subsystem. Instrumentation is performed at the entry or return points of functions such as netif_receive_skb, napi_gro_receive, ip_rcv, and tcp_v4_rcv in the packet receiving path, and dev_queue_xmit in the sending path, to capture the flow and time consumption of packets at each layer of the protocol stack.

[0042] Exception handling branch probes: Attach probes to exception packet loss functions such as kfree_skb, as well as specific condition branches such as ring buffer overflow, memory allocation failure, and checksum error, to accurately record the occurrence time, triggering function, call stack, packet loss reason (such as error code), and associated network flow 5-tuple information (source / destination IP, port, protocol) of each packet loss event.

[0043] Socket layer behavior observation: Attach probes to the kernel functions corresponding to system calls such as tcp_recvmsg and udp_recvmsg, and statistically analyze the application's packet receiving behavior, latency, and abnormal states such as EAGAIN by process ID and socket descriptor.

[0044] Efficient Data Transmission and Aggregation: eBPF programs transmit fine-grained event data collected through Perf Event or BPF Map mechanisms to a proxy service in user space with low overhead. This proxy is responsible for structuring the data according to dimensions such as timestamp, interface index, process ID, and connection identifier, and aligning it with the aforementioned multi-level metrics in time, storing it uniformly in a time-series database to form a globally unified data view.

[0045] Example 2: Feature Construction and Temporal Correlation The feature layer transforms the multi-source heterogeneous data collected by the acquisition layer into features that can be used for analysis and modeling, and establishes cross-layer topological relationships, specifically in the following four aspects: Temporal feature vector construction: Raw metrics are aggregated within a fixed time window (default 30 seconds) to generate a continuous metric sequence with precise timestamps for each network interface, ring buffer queue, network connection, and application process. All raw data and derived features are retained for 7 days for historical backtracking and analysis. It's important to note that the raw metrics are multi-source heterogeneous data collected by the acquisition layer.

[0046] Full-link topology mapping: Utilizing system configuration information and kernel object relationships, a full-link topology mapping is dynamically established from "physical network card → ring buffer → triggering soft interrupt handling task (CPU) → netdev_max_backlog queue → protocol layer queue → socket buffer → application process". This topology is crucial for tracing the source of packet loss points.

[0047] Multidimensional correlation analysis: The function-level events captured by eBPF are associated with the above full-link topology mapping to establish a precise correspondence between "code execution path → queue / interface → process / connection", so that any packet loss event can be accurately located to the specific code branch and the affected business unit.

[0048] Advanced Feature Engineering: Based on the original indicators, a series of advanced features are generated through methods such as sliding window calculation and year-on-year / month-on-month analysis. These features include the sudden growth rate of the indicators, the saturation level of queue utilization, the jitter of processing latency, and the triggering frequency of abnormal branches in eBPF, providing more discriminative information for subsequent intelligent analysis.

[0049] Example 3: Packet Loss Scene Identification and Root Cause Localization The analysis layer, based on the features and relationships generated by the feature layer, uses a hybrid intelligent analysis engine that combines a pre-set rule base and a pre-set machine learning model to automatically identify packet loss scenarios and locate the root cause according to the unique kernel behavior characteristics of the domestic operating system.

[0050] Preset rule base matching: The system scans real-time metrics layer by layer according to the following preset rules to quickly identify common packet loss patterns. Common packet loss types are as follows: Physical layer packet loss: If the packet loss count of the physical network card increases and is accompanied by link protocol errors, it is determined to be a link quality problem; if the network card speed is inconsistent with that of the switch, it is determined to be a duplex / speed mismatch.

[0051] Ring buffer overflow (corresponding) Figure 1 (Ring buffer packet loss): If the receive / send queue utilization is close to 100% for a long period of time, and the overflow count in ethtool statistics continues to increase, it is determined that the hardware queue overflow is caused by a sudden traffic surge.

[0052] Soft interrupt packet loss (corresponding) Figure 1 (In the context of "untimely soft interrupt handling"): If the packet loss count of a CPU in / proc / net / softnet_stat increases, and the eBPF probe detects that the NAPI polling time of that CPU exceeds the threshold or the scheduling delay increases, then it is determined to be a CPU processing bottleneck.

[0053] Backlog queue packet loss (corresponding to) Figure 1 (Netdev_backlog queue overflow and packet loss): If the netdev_max_backlog queue length continuously reaches the kernel parameter limit, and eBPF captures branches dropped due to queue full in functions such as enqueue_to_backlog, then it is determined to be protocol stack entry congestion.

[0054] Protocol stack packet loss: If the TCP retransmission rate increases abnormally, or if packet loss is observed in functions such as tcp_v4_do_rcv due to a zero receive window through eBPF, it is determined to be a protocol layer flow control or memory pressure issue.

[0055] Socket buffer packet loss: If the socket receive buffer usage rate is consistently 100%, and eBPF detects that the application has not called recvmsg for a long time, it is determined that the application is reading slowly, causing the buffer to overflow.

[0056] Process bottleneck packet loss: If eBPF profiling data reveals that the function time of a certain application process when processing network data is much higher than normal, and the upstream queue is backed up as a result, it is determined that there is a performance bottleneck in the application layer code path.

[0057] It is important to know that for complex packet loss scenarios that cannot be covered by the preset rule base or have complicated causes, the system calls preset machine learning models (such as anomaly detection or root cause classification models based on historical data) to perform auxiliary analysis, generate root cause confidence rankings, and provide them to operations and maintenance personnel for reference.

[0058] Example 4: Intelligent Strategy Decision Making Based on the root causes of packet loss determined by the analysis layer, the decision-making and repair layer generates optimal or near-optimal repair suggestions according to the built-in knowledge base of domestic operating system optimization strategies. The strategy knowledge base fully considers the differences in various domestic operating system distributions, kernel versions, and network card drivers, avoiding risks that might be introduced by common parameters. The generated repair suggestions include: Dynamic adjustment of kernel parameters: For example, adjusting the ring buffer size (ethtool -G), setting the CPU affinity of soft interrupts (smp_affinity), modifying interrupt merging parameters, increasing net.core.netdev_max_backlog or socket buffer size (rmem_max / wmem_max).

[0059] CPU and network optimization: For example, using tasksets to bind business processes to specific CPU cores, configuring RPS / RFS (Receive Packet Steering / Flow Steering) to balance multi-core load, and adjusting the number of network card queues, etc.

[0060] Application-side optimization suggestions: For process bottlenecks identified by eBPF, the strategy knowledge base will propose code-level optimization suggestions, such as adding a separate packet receiving thread, optimizing data processing logic, and introducing asynchronous I / O models such as epoll.

[0061] Combination optimization scheme: For complex packet loss scenarios with multiple coupled factors, the engine will generate a set of combination optimization strategies to be executed in sequence, and set safety boundaries (such as maximum value, minimum value, and adjustment step size) for each parameter adjustment action to ensure that the adjustment process is safe and controllable.

[0062] Example 5: Automatic Repair and Closed-Loop Verification After generating repair suggestions, the decision-making and repair layer executes repair actions for networks that are within the scope of the operation and maintenance personnel's authorization or preset whitelist and are not in critical business periods. That is, the system automatically issues and executes parameter modification commands through standard system configuration interfaces (such as sysctl, ethtool, driver parameter files, application configuration files, etc.). For networks that are not within the scope of the operation and maintenance personnel's authorization or preset whitelist, only repair suggestions are output. For networks that are within the scope of the operation and maintenance personnel's authorization or preset whitelist and are in critical business periods, they are configured in read-only mode of observation only without repair.

[0063] During the repair process, a step-by-step adjustment strategy is adopted for key kernel parameters. For example, each adjustment only increases the ring buffer size by a certain percentage, followed by a 3-5 minute observation period. If key indicators (such as packet loss rate and processing latency) improve and no new anomalies appear during this period, the next adjustment is continued; if the indicators deteriorate or an alarm is triggered, the system immediately and automatically rolls back to the previous stable state and records the failure cases.

[0064] After the repair action is completed, the system continuously monitors various indicators of the target link or queue, comparing and evaluating the current key performance indicators (KPIs) such as packet loss rate, processing latency, and queue utilization with the baseline data before the repair. If the improvement effect reaches the preset threshold, the adjusted parameters are fixed in the system configuration; if the threshold is not met, a new round of root cause analysis is automatically triggered to generate a new repair strategy.

[0065] The system records in detail the parameter changes, execution time, and before-and-after effect comparison data for each repair action, forming a complete operation audit log, which provides original materials for subsequent case analysis and knowledge accumulation.

[0066] Example 6: Results Presentation and Knowledge Accumulation The presentation layer communicates with the acquisition layer, feature layer, analysis layer, and decision-making / remediation layer. It provides a unified, visual operation and maintenance interface, intuitively displaying packet loss detection results, root cause analysis conclusions, remediation execution processes, and final verification results through link topology diagrams, key indicator trend charts, and event timelines. It supports flexible filtering and data drilling by multiple dimensions, including host, network interface card, service tag, process name, and network connection. eBPF's observation results support drill-down to the function-level hotspot call stack, facilitating in-depth performance analysis for developers.

[0067] Meanwhile, the presentation layer has a built-in alarm engine that notifies operations and maintenance personnel in real time via email, SMS, Webhook, and other means for events such as packet loss exceeding dynamic thresholds, failure of automatic repair tasks, and rollback of parameter adjustments, ensuring that abnormal situations are addressed promptly.

[0068] The knowledge layer communicates with both the presentation and analysis layers. It automatically transforms each successful or failed packet loss handling case (including scenario characteristics, root causes, implemented repair strategies, and final verification results) into structured knowledge and stores it in a centralized knowledge base. The system periodically uses these newly stored cases to incrementally train and iteratively optimize the pre-set rule base and pre-set machine learning models, thereby continuously improving the accuracy of identifying and repairing new and similar faults, forming a continuously self-optimizing intelligent operation and maintenance closed loop.

[0069] Example 7: User Flow like Figure 3 As shown, the system of this invention is deployed on the Galaxy Kylin Advanced Server Operating System V10, supports single-machine deployment mode, and is responsible for local network indicator collection, eBPF observation and repair execution.

[0070] Operations personnel can log in to the target server via SSH and run net_drop_system directly in the terminal to achieve a closed loop of local indicator collection, packet loss analysis, and repair execution.

[0071] User flow: After the operations and maintenance personnel specify the network interface to be monitored (e.g., `--iface eth0`) and the critical service port (e.g., `--port 80`), the tool is launched, and the system continuously collects the packet loss rate of the interface. When the packet loss rate exceeds the 0.1% threshold for 3 consecutive seconds, the eBPF fine-grained observation and intelligent analysis module is automatically triggered. The terminal displays the packet loss events captured by eBPF in real time (process ID, call stack, error code, etc.), and outputs the main cause classification (buffer overflow / driver error / protocol stack problem) and optimization suggestions after the analysis is completed.

[0072] By default, the system only outputs repair suggestions for operations and maintenance personnel to manually execute. In a test environment or when using the `--apply` parameter, the system can automatically perform parameter adjustments within safety boundaries (such as progressively increasing `netdev_max_backlog`) and output a comparison before and after the adjustment. After the repair, operations and maintenance personnel can run the tool again to collect metrics and compare key indicators such as packet loss rate, queue depth, and processing latency before and after the repair to verify the repair effect.

[0073] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A network link anomaly intelligent analysis and repair system under a domestically developed operating system, characterized in that, It includes a data acquisition layer, a feature layer, an analysis layer, a decision repair layer, a display layer, and a knowledge layer. The data acquisition layer, feature layer, analysis layer, and decision repair layer are sequentially and communicatively connected. The data acquisition layer, feature layer, analysis layer, and decision repair layer are all communicatively connected to the display layer. The knowledge layer is communicatively connected to both the display layer and the analysis layer. The acquisition layer is used to collect multi-source heterogeneous data on network operation status from multiple dimensions. This includes periodically collecting network performance and packet loss indicators of the physical network card layer, network card ring buffer layer, soft interrupt and NAPI layer, netdev_max_backlog layer, protocol stack layer and socket buffer layer by reading / proc, sys file system interface and system command calls. It also uses a dual mechanism based on eBPF technology, which prioritizes static tracepoint adaptation and uses dynamic function symbol resolution as a fallback. Customized eBPF programs are mounted at key functions and tracepoints of the network subsystem of the domestic operating system kernel to capture the flow of data packets in each layer of the protocol stack, abnormal packet loss events and socket layer behavior data. The feature layer is used to perform time alignment and correlation on the multi-source heterogeneous data acquired by the acquisition layer, transforming it into features that can be used for analysis and modeling, and establishing cross-level correlation relationships. The analysis layer is used to automatically identify typical packet loss scenarios and locate the root cause of packet loss based on the features and relationships generated by the feature layer and an analysis engine that combines a preset rule base and a preset machine learning model. The decision repair layer is used to generate and execute optimal or suboptimal repair suggestions based on the built-in strategy knowledge base specifically designed for tuning domestic operating systems after the analysis layer determines the root cause of packet loss. The presentation layer is used to provide a unified visual operation and maintenance interface, which displays packet loss detection results, root cause location conclusions, repair execution process and verification results in the form of link topology diagram, key indicator trend diagram and event timeline. It supports multi-dimensional filtering and data drilling by host, network card interface, service tag, process name and network connection. The knowledge layer is used to automatically transform each successful or failed packet loss handling case into structured knowledge and deposit it into a centralized knowledge base. The newly deposited cases are used periodically to perform incremental training and iterative optimization on the preset rule base and preset machine learning model.

2. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 1, characterized in that, Data captured based on eBPF technology includes: Attach probes to the entry or return point of the netif_receive_skb, napi_gro_receive, ip_rcv, tcp_v4_rcv functions in the packet receiving path, and the dev_queue_xmit function in the sending path to capture the flow and time consumption of packets in each layer of the protocol stack. Mount probes at the kfree_skb abnormal packet loss function and at the condition branches of ring buffer overflow, memory allocation failure, and checksum error to record the time of packet loss event, triggering function, call stack, packet loss cause and its associated network flow quintuple information; Attach probes to the kernel functions corresponding to the tcp_recvmsg and udp_recvmsg system calls to collect statistics on the application's packet receiving behavior, latency, and abnormal status by process ID and socket descriptor.

3. The intelligent analysis and repair system for network link anomalies under a domestically developed operating system according to claim 1, characterized in that, The establishment of cross-level associations in the feature layer includes: The raw metrics are aggregated using a fixed time window to generate a continuous sequence of metrics with precise timestamps for each network interface, ring buffer queue, network connection, and application process. By utilizing system configuration information and kernel object relationships, a full-link topology mapping is dynamically established from "physical network card → ring buffer → trigger soft interrupt handling task → netdev_max_backlog queue → protocol layer queue → socket buffer → application process"; Associating eBPF-captured function-level events with the end-to-end topology mapping establishes a precise correspondence between "code execution path → queue / interface → process / connection".

4. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 1, characterized in that, The analysis layer uses a preset rule base to identify packet loss scenarios, including: Physical layer packet loss determination: If the packet loss count of the physical network card increases and is accompanied by link protocol errors, it is determined to be a link quality problem; if the network card speed is inconsistent with that of the switch, it is determined to be a duplex / speed mismatch. Ring buffer overflow determination: If the receive / transmit queue is full and the overflow count continues to increase, it is determined that the hardware queue overflow is caused by a sudden traffic surge. Soft interrupt packet loss determination: If the packet loss count of a certain CPU increases during the soft interrupt phase and the NAPI polling time exceeds the threshold, it is determined to be a CPU processing bottleneck. Backlog queue packet loss determination: If the netdev_max_backlog queue length continuously reaches the kernel parameter limit and branches dropped due to the queue being full are captured, it is determined to be protocol stack entry congestion; Protocol stack packet loss determination: If the TCP retransmission rate increases abnormally or packet loss is observed due to the receive window being zero, it is determined to be a protocol layer flow control or memory pressure issue. Socket buffer packet loss determination: If the socket receive buffer usage rate is consistently 100% and the application has not called recvmsg for a long time, it is determined that the application is reading slowly, causing the buffer to overflow.

5. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 1, characterized in that, The decision-making repair layer generates repair suggestions including: Dynamic adjustment of kernel parameters: adjust the ring buffer size, set the CPU affinity of soft interrupts, modify interrupt merging parameters, increase net.core.netdev_max_backlog or socket buffer size; CPU and network optimization: bind business processes to specific CPU cores, configure RPS / RFS to balance multi-core load, and adjust the number of network card queues; Application-side optimization suggestions: To address the process bottleneck identified by eBPF, we propose adding a packet receiving thread, optimizing data processing logic, and introducing an asynchronous I / O model; Combinatorial optimization scheme: For complex packet loss scenarios with multiple coupled factors, a combination optimization strategy is generated and executed sequentially, and a safety boundary is set for the adjustment action of each parameter.

6. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 1, characterized in that, The decision-making repair layer implements repair suggestions including: For network server nodes that are included in the operation and maintenance authorization scope and are within the preset whitelist, and are not during peak hours of critical business, parameter modification commands are automatically issued and executed through sysctl, ethtool, driver parameter files, and application configuration files. A step-by-step adjustment strategy is adopted for key kernel parameters. Each adjustment only increases the parameter value by a preset proportion. After the adjustment, the observation period begins. If the key indicators improve and there are no new anomalies, the next adjustment is continued. If the indicators deteriorate or an alarm is triggered, the system will immediately and automatically roll back to the previous stable state.

7. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 6, characterized in that, Also includes: After the repair action is completed, the decision repair layer continuously monitors various indicators of the target link or queue, compares and evaluates the current key performance indicators with the baseline data before the repair, and if the improvement effect reaches the preset threshold, the adjusted parameters will be fixed; if the target is not met, a new round of root cause analysis will be automatically triggered.

8. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 7, characterized in that, Also includes: The decision-making and repair layer meticulously records the parameter changes, execution time, and before-and-after effect comparison data for each repair action, forming a complete operation audit log that provides raw materials for subsequent case analysis and knowledge accumulation.

9. A network link anomaly intelligent analysis and repair system under a domestic operating system according to claim 6, characterized in that, The observation period lasts 3-5 minutes.

10. The intelligent analysis and repair system for network link anomalies under a domestic operating system according to claim 1, characterized in that, The system supports stand-alone deployment mode.

Images