A network reliability enhancement method in an industrial control scenario

Abstract

The application relates to a network reliability enhancement method in an industrial control scene, which comprises the following steps: S100, starting a kernel monitoring thread for traversing all UDP sockets and TCP sockets in a system in a kernel; S200, collecting a corresponding sk_wmem_alloc parameter value of each traversed socket in real time, and judging whether the parameter value exceeds a dynamic threshold; S300, if the parameter value of the same socket exceeds the dynamic threshold for three times in succession, marking the socket as a fault socket, marking a network card sending queue corresponding to the fault socket as a fault sending queue, and triggering a fault processing flow; and S400, executing the fault processing flow. The main purpose of the application is to solve the memory leakage problem caused by the fact that the socket sending queue content cannot be destroyed under the condition that the CPU is busy and cannot process the network card sending completion interrupt in time under the condition of high interruption load and high network IO.

Description

Technical Field

[0001] This invention relates to the field of industrial control technology, and more specifically to a method for enhancing network reliability in industrial control scenarios. Background Technology

[0002] In industrial control scenarios, to ensure equipment reliability, network interface cards (NICs) with multi-queue functionality are often selected. However, since multiple queues of the NIC are under the same NIC control chip, when the interrupt load is high and the network pressure is heavy, the NIC may complete sending a data packet, but the interrupt notifying the CPU may be discarded. This causes some data packets to remain in the protocol stack, preventing the corresponding network socket's send queue from being released. Consequently, the application cannot send data through that socket, ultimately leading to abnormal network data transmission in the industrial control program and causing the system state to become uncontrollable.

[0003] To solve the above problems, technicians developed several solutions:

[0004] A domestic patent with announcement number CN110166323B discloses a method to improve the reliability of a blockchain network. The patent implements a blockchain network reliability method that uses a GRPC link in the workshop to determine whether the network is reliable. If the GRPC link is unreliable, it uses TCP to send data packets to determine whether the network is reliable.

[0005] A document with domestic publication number CN115270368A discloses a method, apparatus, and computer equipment for evaluating the reliability of network systems based on simulation. The document describes how to evaluate the reliability of a target network through simulation results.

[0006] A domestic patent with announcement number CN114358248B discloses a method, medium, and device for measuring the redundancy reliability of deep neural networks. The patent discloses a method for measuring the redundancy reliability of deep neural networks, which involves weight-based clustering of network layer nodes, using geometric centers to identify the center of each layer, constructing a reliability calculation model based on a fault injection strategy, and extracting redundancy reliability based on an adaptive convergence strategy. Specifically, it includes steps such as reading model parameters, weight clustering, marking redundant nodes, and judging redundancy reliability through the model.

[0007] A domestic patent with announcement number CN109842895 discloses a network reliability configuration method, information transmission method and device and system. This patent discloses a network reliability configuration method applied to the policy control function (PCF) entity. The PCF entity first obtains reliability request information containing UE service flow reliability indicators generated by the application function (AF) entity, then extracts the indicators to determine the network reliability guarantee requirements of the service flow, and finally determines the redundant transmission strategy used to instruct the session management function (SMF) entity to configure redundant transmission paths.

[0008] A domestic patent with publication number CN115066871B discloses a system and method for improving network reliability. The patent discloses a network control method implemented by the main network controller in a controller cluster. The main network controller establishes an IGP adjacency relationship with one or more network elements in the network through an IGP agent and announces its own controller information (included in the controller type TLV). It also identifies communication faults by detecting the heartbeat stoppage of the auxiliary network controller, and then announces updated controller information reflecting information changes to the network elements.

[0009] The aforementioned existing technology uses multiple single-queue network interface card (NIC) controllers, with each NIC controller controlling one NIC, thereby avoiding the problem mentioned above. However, this method has drawbacks, mainly that a single embedded SoC chip suitable for industrial control scenarios can control a limited number of NIC controllers. Furthermore, multiple NIC controllers mean higher costs, more complex internal control logic within the SoC chip, and are not feasible for most industrial control scenarios. Summary of the Invention

[0010] The main objective of this invention is to provide a method for enhancing network reliability in industrial control scenarios, in order to solve the memory leak problem caused by the inability to destroy the socket transmission queue contents in Linux systems under high interrupt load and high network I / O scenarios, due to the CPU being too busy to handle network card transmission completion interrupts in a timely manner.

[0011] To achieve the above objectives, this invention provides a method for enhancing network reliability in industrial control scenarios, applied to a Linux system equipped with a multi-queue network interface card, comprising the following steps:

[0012] S100: Starts a kernel listener thread in the kernel with a period of 1 second. The kernel listener thread is used to traverse all UDP sockets and TCP sockets in the system.

[0013] S200: For each traversed socket, collect its corresponding sk_wmem_alloc parameter value in real time, and determine whether the sk_wmem_alloc parameter value exceeds the dynamic threshold. The dynamic threshold is a value calculated based on the system-preset net.core.wmem_max parameter.

[0014] S300: If the sk_wmem_alloc parameter value of the same socket exceeds the dynamic threshold three times in a row, the socket is marked as a faulty socket, and the network card transmission queue corresponding to the faulty socket is marked as a faulty transmission queue, triggering the fault handling process.

[0015] S400: Execute fault handling procedure.

[0016] Preferably, step S400 includes the following steps:

[0017] S410: Retrieves all sk_buff structures (abbreviated as skb) associated with the faulty socket;

[0018] S420: Adjust the send queue mapping of all sk_buff structures and migrate them to the send queues in the system that are not marked as faulty.

[0019] S430: Disable interrupts in the fault transmission queue;

[0020] S440: Release the resources corresponding to all sk_buff structures in the fault transmission queue, clear the addresses of all DMA descriptors in the fault transmission queue, and clear the TX_USED status bit of the ctrl field of all DMA descriptors.

[0021] S450: Interrupt that reopens the fault transmission queue;

[0022] S460: Clear the fault status flags of the fault transmission queue and the fault status flags of the fault socket.

[0023] More preferably, the specific rules for adjusting the send queue mapping relationship of all sk_buff structures in step S420 are as follows:

[0024] Get the index value of the fault sending queue, queue_index. If queue_index is 0, then map all skb sending queues to the sending queue with index value 1; if queue_index is 1, then map all skb sending queues to the sending queue with index value 0.

[0025] More preferably, the specific implementation of releasing the resources corresponding to all sk_buff structures in the fault transmission queue in step S440 is as follows: obtain the hardware transmission queue structure of the fault transmission queue through macb->queues[queue_index] in the network card private structure macb, traverse all macb_tx_skb structures (referred to as tx_skb) in the hardware transmission queue with index values ​​from 0 to tx_ring_size, and release the resources corresponding to each tx_skb in turn.

[0026] More preferably, tx_ring_size is the size of the ring buffer of the hardware transmit queue in the MACB multi-queue network card. This value is preset by the hardware design parameters of the network card and is read and fixed when the Linux system kernel loads the network card driver.

[0027] Preferably, the specific implementation of traversing all UDP sockets in step S100 is as follows:

[0028] By using the global variable udp_table in the kernel, all udp_hslot hash buckets are traversed, and then the hlist_head linked list within each udp_hslot hash bucket is traversed. Each node in the hlist_head linked list is a sock node corresponding to a UDP socket.

[0029] Preferably, the specific implementation of traversing all TCP sockets in step S100 is as follows:

[0030] By traversing the global tcp_hashinfo structure in the kernel, the inet_ehash_bucket, inet_bind_hashbucket, and inet_listen_hashbucket hash buckets contained therein are used to retrieve the sock nodes corresponding to all TCP sockets from the three hash buckets.

[0031] Preferably, the specific calculation method for the dynamic threshold in step S200 is: net.core.wmem_max / 2-1000, where net.core.wmem_max is the maximum value parameter of the socket send buffer in the Linux system.

[0032] Preferably, step S300 further includes:

[0033] If the sk_wmem_alloc parameter value of the same socket does not exceed the dynamic threshold three times consecutively, the over-threshold warning count of the socket is reset to 0, and the socket is determined to be a normal socket.

[0034] The beneficial effects of this invention are as follows:

[0035] This application implements a multi-queue network interface card (NIC) platform based on the Linux system, which can be used to enhance network reliability for UDP and TCP (UDP is mostly used for network transmission in industrial control scenarios). By monitoring all UDP and TCP sockets, when the `sk_wmem_alloc` value exceeds the threshold three times consecutively, the send queue used by that socket is marked as a faulty queue. Network reliability is improved by adjusting the send queue to a non-faulty queue and clearing all resources in the faulty queue. This allows for real-time and accurate detection of UDP and TCP socket anomalies, preventing fault concealment and propagation, achieving fault self-healing, ensuring the business continuity of industrial control systems such as DCS systems, and reducing maintenance costs and losses caused by faults. Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] Example 1

[0038] This embodiment provides a method for enhancing network reliability in industrial control scenarios, applied to a Linux system equipped with a multi-queue network interface card, including the following steps:

[0039] S100: Starts a kernel listener thread in the kernel with a period of 1 second. The kernel listener thread is used to traverse all UDP sockets and TCP sockets in the system.

[0040] In this step, the specific implementation of traversing all UDP sockets is as follows:

[0041] By using the global variable udp_table in the kernel, all udp_hslot hash buckets are traversed, and then the hlist_head linked list within each udp_hslot hash bucket is traversed. Each node in the hlist_head linked list is a sock node corresponding to a UDP socket.

[0042] The specific implementation method for traversing all TCP sockets is as follows:

[0043] By traversing the global tcp_hashinfo structure in the kernel, the inet_ehash_bucket, inet_bind_hashbucket, and inet_listen_hashbucket hash buckets contained therein are used to retrieve the sock nodes corresponding to all TCP sockets from the three hash buckets.

[0044] S200: For each traversed socket, collect the corresponding sk_wmem_alloc parameter value in real time, and determine whether the sk_wmem_alloc parameter value exceeds the dynamic threshold. The dynamic threshold is a value calculated based on the system-preset net.core.wmem_max parameter.

[0045] In step S200, the dynamic threshold is calculated as follows: net.core.wmem_max / 2-1000, where net.core.wmem_max is the maximum value parameter of the socket send buffer in the Linux system.

[0046] S300: If the sk_wmem_alloc parameter value of the same socket exceeds the dynamic threshold three times consecutively, the socket is marked as a faulty socket, and the network card transmission queue corresponding to the faulty socket is marked as a faulty transmission queue, triggering the fault handling process.

[0047] In step S300, the following method is also included: if the sk_wmem_alloc parameter value of the same socket does not exceed the dynamic threshold three times consecutively, the over-threshold warning count of the socket is reset to 0, and the socket is determined to be a normal socket.

[0048] S400: Execute fault handling procedure.

[0049] Step S400 includes the following steps:

[0050] S410: Retrieves all sk_buff structures (abbreviated as skb) associated with the faulty socket.

[0051] S420: Adjust the send queue mapping of all sk_buff structures and migrate them to the send queues in the system that are not marked as faulty.

[0052] The specific rules for adjusting the sending queue mapping relationship of all sk_buff structures in step S420 are as follows: obtain the index value queue_index of the faulty sending queue. If queue_index is 0, then map the sending queue of all skb to the sending queue with index value 1; if queue_index is 1, then map the sending queue of all skb to the sending queue with index value 0.

[0053] S430: Disable interrupts for the fault transmission queue.

[0054] S440: Release the resources corresponding to all sk_buff structures in the fault transmission queue, clear the addresses of all DMA descriptors in the fault transmission queue, and clear the TX_USED status bit of the ctrl field of all DMA descriptors.

[0055] In step S440, the specific implementation of releasing the resources corresponding to all sk_buff structures in the fault transmission queue is as follows: obtain the hardware transmission queue structure of the fault transmission queue through macb->queues[queue_index] in the network card private structure macb, traverse all macb_tx_skb structures (referred to as tx_skb) in the hardware transmission queue with index values ​​from 0 to tx_ring_size, and release the resources corresponding to each tx_skb in turn.

[0056] Among them, tx_ring_size is the size of the ring buffer of the hardware transmit queue in the MACB multi-queue network card. This value is preset by the hardware design parameters of the network card and is read and fixed when the network card driver is loaded in the Linux system kernel.

[0057] S450: Interrupt that reopens the fault transmission queue;

[0058] S460: Clear the fault status flags of the fault transmission queue and the fault status flags of the fault socket.

[0059] Example 2

[0060] This embodiment is based on a specific implementation scenario of Embodiment 1.

[0061] I. Implementation Prerequisites

[0062] Hardware preparation:

[0063] The device uses Phytium E2000Q equipment and MACB multi-queue network card. This hardware combination is a common configuration in industrial control scenarios and is adapted to the needs of multi-queue data transmission.

[0064] II. Kernel Listener Thread Implementation

[0065] 1. Basic thread configuration

[0066] Create a kernel-level listening thread with an execution cycle of 1 second to ensure that all UDP and TCP sockets in the system are traversed once per second, thereby achieving real-time monitoring of socket status.

[0067] 2. UDP socket traversal

[0068] Using the Linux kernel global variable udp_table, we first traverse all udp_hslot hash buckets, and then traverse the hlist_head linked list within each udp_hslot hash bucket. Each node in this linked list is a sock node corresponding to a UDP socket. In this way, we can obtain all UDP sockets in the system.

[0069] 3. TCP socket traversal

[0070] By leveraging the Linux kernel's global tcp_hashinfo structure, and traversing its three hash buckets—inet_ehash_bucket, inet_bind_hashbucket, and inet_listen_hashbucket—the system extracts the sock nodes corresponding to all TCP sockets from each hash bucket, thus achieving comprehensive coverage listening for both UDP and TCP sockets.

[0071] 4. Parameter Acquisition and Threshold Determination

[0072] For each socket node traversed, its sk_wmem_alloc parameter value is collected in real time (this parameter represents the number of memory bytes occupied by the socket send buffer).

[0073] A dynamic threshold is set, calculated as net.core.wmem_max / 2-1000, where net.core.wmem_max is the maximum value parameter of the socket send buffer preset by the Linux system. The threshold is designed to be slightly lower than the theoretical critical value to improve the accuracy of judgment in actual scenarios.

[0074] Compare the collected sk_wmem_alloc value with the dynamic threshold: if sk_wmem_alloc is greater than the threshold, mark one warning; if it is not greater than the threshold, the socket is determined to be normal, and the over-threshold warning count of the socket is reset to 0.

[0075] 5. Fault marking logic

[0076] The warning count for each socket is accumulated. If the value of the sk_wmem_alloc parameter of the same socket exceeds the dynamic threshold three times in a row (i.e., a warning is detected in 3 consecutive thread cycles), the socket is marked as a faulty socket and the relevant logs of the faulty socket are printed (including socket identifier, current value of sk_wmem_alloc, etc.).

[0077] At the same time, by using the transmission queue information associated with the faulty socket, the network card transmission queue used by the socket is marked as a faulty transmission queue, triggering subsequent fault handling procedures.

[0078] III. Implementation of Fault Handling Functions

[0079] Add a queue_error_handle fault handling function to the macb network card driver core file macb_main.c:

[0080] 1. Obtain core information related to the fault

[0081] It receives a faulty socket (sock) as input parameter and obtains the first sk_buff (skb for short, which is the core structure of the socket send buffer in the kernel) in the linked list through the sk_write_queue structure of sock.

[0082] By parsing the queue_mapping variable of the skb, the queue_index value corresponding to the sending queue of the skb can be obtained, and the specific identifier of the faulty sending queue can be clearly identified.

[0083] 2. skb queue migration

[0084] Iterate through all skb in the faulty socket sk_write_queue linked list and adjust the send queue mapping relationship according to fixed rules:

[0085] If the fault sending queue index queue_index is 0, then all skbs are migrated to the non-fault sending queue with index 1 through the skb_set_queue_mapping interface;

[0086] If the faulty sending queue index queue_index is 1, then the data will be migrated to the non-faulty sending queue with index 0 to ensure that data transmission is not interrupted.

[0087] 3. Fault queue interrupt control

[0088] Disable interrupts in the fault transmission queue: Obtain the interrupt identifier of the fault queue through the network card's private structure macb, and perform an interrupt disabling operation to prevent the interrupt loss problem from escalating further.

[0089] After the fault is resolved, the interrupt for the fault queue is reopened to restore the queue's normal operating capability.

[0090] 4. Resource Release and Cleanup

[0091] First, obtain the network interface device (net_device) through the dev structure of skb. Then, extract the network interface private structure macb through the netdev_priv function. Finally, obtain the hardware transmission queue structure corresponding to the fault transmission queue through macb->queues[queue_index].

[0092] Iterate through all macb_tx_skb (tx_skb for short) in the hardware transmit queue with index values ​​from 0 to tx_ring_size:

[0093] Release the memory resources corresponding to each tx_skb in turn;

[0094] Clear the addr field (i.e., DMA address) of each tx_skb descriptor in turn;

[0095] Clear the TX_USED status bit of the ctrl field of each tx_skb descriptor in turn.

[0096] Among them, tx_ring_size is the size of the ring buffer of the hardware transmit queue in the MACB multi-queue network card. This value is preset by the hardware design parameters of the network card and is read and fixed when the network card driver is loaded in the Linux system kernel.

[0097] 5. Status Reset

[0098] Clear the fault status flag of the send queue corresponding to queue_index;

[0099] Clear the fault status flag of the faulty socket;

[0100] Print the fault handling completion log (including the number of skb released, fault queue index, recovery time, etc.).

[0101] IV. System Deployment and Testing Verification

[0102] 1. Kernel compilation and deployment

[0103] The newly added kernel listener thread code and queue_error_handle fault handling function code are integrated into the corresponding kernel source code files and compiled to generate a new kernel image.

[0104] Reboot the Phytium E2000Q device, select the newly compiled kernel to enter the system, and complete the deployment of the technical solution.

[0105] 2. Testing and Verification Process

[0106] Interrupt load generation: Use the FIO tool to generate more than 10,000 interrupts per second to simulate high interrupt load in industrial control scenarios.

[0107] Network stress testing: Use the iperf tool to perform multi-process UDP stress testing, fully utilize network bandwidth, and simulate high network I / O scenarios.

[0108] Long-term stability testing: Continuous 24 / 7 stress testing and monitoring of system logs.

[0109] Results Verification: It was observed that a certain iperf test stream automatically recovered after being interrupted for 3 seconds. Checking the logs revealed the corresponding fault socket identification log and fault clearing log, verifying the effectiveness of the technical solution.

[0110] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for enhancing network reliability in industrial control scenarios, characterized in that, For Linux systems equipped with multi-queue network cards, the following steps are included: S100: Starts a kernel listener thread in the kernel with a period of 1 second. The kernel listener thread is used to traverse all UDP sockets and TCP sockets in the system. S200: For each traversed socket, collect its corresponding sk_wmem_alloc parameter value in real time, and determine whether the sk_wmem_alloc parameter value exceeds the dynamic threshold. The dynamic threshold is a value calculated based on the system-preset net.core.wmem_max parameter. S300: If the sk_wmem_alloc parameter value of the same socket exceeds the dynamic threshold three times in a row, the socket is marked as a faulty socket, and the network card transmission queue corresponding to the faulty socket is marked as a faulty transmission queue, triggering the fault handling process. S400: Execute the fault handling procedure; Step S400 includes the following steps: S410: Retrieves all sk_buff structures associated with the faulty socket; S420: Adjust the sending queue mapping relationship of all sk_buff structures and migrate them to the sending queues in the system that are not marked as faulty. The specific rules for adjusting the sending queue mapping relationship of all sk_buff structures in step S420 are as follows: obtain the index value queue_index of the faulty sending queue. If queue_index is 0, then map the sending queues of all skb structures to the sending queue with index value 1; if queue_index is 1, then map the sending queues of all skb structures to the sending queue with index value 0. S430: Disable interrupts in the fault transmission queue; S440: Release the resources corresponding to all sk_buff structures in the fault transmission queue, clear the addresses of all DMA descriptors in the fault transmission queue, and clear the TX_USED status bit of the ctrl field of all DMA descriptors; The specific implementation of releasing the resources corresponding to all sk_buff structures in the fault transmission queue in step S440 is as follows: Obtain the hardware transmission queue structure of the fault transmission queue through macb->queues[queue_index] in the network card private structure macb, traverse all macb_tx_skb structures with index values ​​from 0 to tx_ring_size in the hardware transmission queue, and release the resources corresponding to each tx_skb in turn; S450: Interrupt that reopens the fault transmission queue; S460: Clear the fault status flags of the fault transmission queue and the fault status flags of the fault socket.

2. A method for enhancing network reliability in an industrial control scenario according to claim 1, characterized in that, tx_ring_size is the size of the ring buffer for the hardware transmit queue in a MACB multi-queue network card. This value is preset by the network card's hardware design parameters and is read and fixed when the Linux system kernel loads the network card driver.

3. A method for enhancing network reliability in an industrial control scenario according to claim 1, characterized in that, The specific implementation of traversing all UDP sockets in step S100 is as follows: By using the global variable udp_table in the kernel, all udp_hslot hash buckets are traversed, and then the hlist_head linked list within each udp_hslot hash bucket is traversed. Each node in the hlist_head linked list is a sock node corresponding to a UDP socket.

4. A method for enhancing network reliability in an industrial control scenario according to claim 1, characterized in that, The specific implementation method for traversing all TCP sockets in step S100 is as follows: By traversing the global tcp_hashinfo structure in the kernel, the inet_ehash_bucket, inet_bind_hashbucket, and inet_listen_hashbucket hash buckets contained therein are used to retrieve the sock nodes corresponding to all TCP sockets from the three hash buckets.

5. A method for enhancing network reliability in an industrial control scenario according to claim 1, characterized in that, Step S300 also includes: If the sk_wmem_alloc parameter value of the same socket does not exceed the dynamic threshold three times consecutively, the over-threshold warning count of the socket is reset to 0, and the socket is determined to be a normal socket.