Method, system and apparatus for supporting two network interface cards and supporting diversified loads by TCP user mode protocol stack, and application thereof

Abstract

The present application relates to the technical field of system communications, and particularly relates to a method, system and apparatus for supporting two network interface cards and supporting diversified loads by a TCP user mode protocol stack. The method comprises: SPDK transport layer addition and TCP user mode protocol stack transformation, involving: adding a new transport layer to an SPDK to support a TCP, transforming a TCP user mode protocol stack, and putting a network interface card packet receiving processing function into a poller of the SPDK, so as to support two network interface cards; configuration and optimization, involving: alternately sending data packets by means of network interface card sorting, so as to realize linear growth in a storage bandwidth and an IOPS; abnormality handling and fault switching, involving: when a certain network interface card fails to send a packet for a long period of time, determining that the network interface card is abnormal, and after timeout, triggering a callback function to re-issue an IO request to another available network interface card; and monitoring and maintenance. In the present application, the performance of the user mode protocol stack with two network interface cards can increase linearly, realizing linear growth in terms of performance.

Description

A method, system, device and application of a TCP user-mode protocol stack supporting dual network cards and diversified loads

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on December 13, 2023, with application number 202311710215.8 and invention name “A method, system and device for a TCP user-mode protocol stack supporting dual network cards and diversified loads”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of system communication technology, and in particular to a method, system, device and application thereof for a TCP user-mode protocol stack to support dual network cards and diversified loads. Background Art

[0004] Computing, storage, and networking are three key components of cloud computing. As CPU computing power continues to increase, network performance is also improving, and the demand for high-performance storage is becoming increasingly urgent. Many older models have multiple 25Gb / s network cards, and with the continuous iteration of new models, this number is increasing. To address hardware bottlenecks, the NVMe protocol emerged. However, high-performance cloud disks are in high demand and have a wide coverage in cloud computing. NVMe hard drives alone cannot meet actual user needs.

[0005] To address low network transmission performance, NVMe over RDMA is a good option. It leverages RDMA's high performance and zero-copy technology to achieve high-performance I / O transmission and ultra-high cloud disk performance. However, RDMA has its own technical limitations. For cross-AZ access in cloud computing, network performance degrades, making it unsuitable for real-world production environments. TCP, on the other hand, is suitable for storage transmission in the kernel protocol stack, but its performance is poor.

[0006] Therefore, cloud computing storage scenarios urgently need a high-performance and highly compatible network transmission protocol. Summary of the Invention

[0007] In order to achieve the above objectives, the present application provides a method, system, device and application of a TCP user-mode protocol stack that supports dual network cards and diversified loads, so as to solve the problem of difficulty in achieving low latency, high concurrency and high performance when accelerating the network in scenarios such as storage clusters, virtualization, GPU AI, etc.

[0008] This application proposes a method for a TCP user-mode protocol stack to support dual network cards and diverse loads, including:

[0009] Adding the SPDK transport layer and modifying the TCP user-mode protocol stack: Adding a new transport layer to SPDK to support the TCP protocol; modifying the TCP user-mode protocol stack and placing the network card packet processing function in the SPDK poller to support dual network cards;

[0010] Configuration and optimization: By sending data packets alternately through network cards, we can achieve linear growth in storage bandwidth and IOPS.

[0011] Exception handling and failover: When a network card fails to send packets for a long time, it is judged as an exception and a callback function is triggered after the timeout to resend the IO request to another available network card;

[0012] Monitoring and maintenance: monitoring network status, including the network card's packet sending and receiving status and network bandwidth utilization; regularly maintaining and optimizing the system to ensure system stability and performance.

[0013] Optionally, when there is a need to independently select a network card for IO processing, one of the dual network cards is selected as the main network card and the other as the backup network card. The main network card sends and receives network data packets. When the main network card fails, it switches to the backup network card to send data packets.

[0014] Optionally, when a network card fails to send packets for a long time, it is judged as abnormal.

[0015] The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

[0016] Optionally, when a network card fails to send packets for a long time, it is judged as abnormal.

[0017] IO will attempt to be sent from the non-abnormal network card.

[0018] The embodiment of the present application provides a data-driven front-end resource loading optimization method. Compared with the prior art, its beneficial effects are:

[0019] The user-mode protocol stack of this application is currently the only protocol stack that supports dual network cards, and the network card performance increases linearly; in actual production environment testing and verification, the bandwidth of a single 25Gb / s single network card is about 3GB / s, and the IOPS is only 70W IOPS. The storage bandwidth of the 2x25Gb / s mode can reach 6GB / s, and the IOPS can reach 139.5W. The experiment proves that the performance in the dual network card mode achieves linear growth; the TCP user-mode protocol stack is not only close to RDMA performance, but also supports Kernel TCP while supporting itself, with very good compatibility, which can well meet the network transmission needs in cloud computing; in cloud computing scenarios, there are many old models with multiple 25Gb / s network cards. With the continuous iteration of models, there will be more and more old models. In order to make good use of these models and reduce enterprise costs, the TCP user-mode protocol stack needs to support multiple network cards to achieve linear performance growth, so that these old models can play a greater value and realize resource reuse.

[0020] On the other hand, the present application proposes a system for TCP user-mode protocol stack supporting dual network cards and supporting diversified loads, which is applied to the above-mentioned method for TCP user-mode protocol stack supporting dual network cards and supporting diversified loads. The system includes: a storage cluster,

[0021] The storage cluster includes: a storage service and several storage nodes. The storage service includes: a file service, an object service and a block service. The file service, the object service and the block service are respectively provided with a transport layer, and the file service, the object service and the block service are respectively electrically connected to the storage nodes.

[0022] Optionally, it further includes: a file server, which is electrically connected to the storage cluster, and the file server includes: a TCP user mode protocol stack and several Ethernet interfaces for implementing network connection and data transmission functions.

[0023] Optionally, the file server is electrically connected to the file service, the object service and the block service respectively.

[0024] Optionally, several of the Ethernet interfaces operate simultaneously.

[0025] On the other hand, the present application also proposes a device for a TCP user-mode protocol stack supporting dual network cards and supporting diversified loads. When the device is running, it executes the above-mentioned method for a TCP user-mode protocol stack supporting dual network cards and supporting diversified loads.

[0026] On the other hand, this application also provides a method for TCP user-mode protocol stack to support dual network cards and diversified loads for application in cloud host virtualization scenarios, container scenarios, GPU AI scenarios and distributed storage scenarios.

[0027] It is understandable that the above-mentioned method, system and device for supporting dual network cards and diversified loads of the TCP user mode protocol stack have the same beneficial effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0029] FIG1 is a schematic diagram of a storage scenario supported by SPDK and TCP user-mode protocol stacks in an embodiment of the present application;

[0030] FIG2 is an IO flow chart before a failure of a dual network card occurs in an embodiment of the present application;

[0031] FIG3 is a switching flow chart after a single network card fails in an embodiment of the present application;

[0032] FIG4 is a schematic diagram of a dual network card active / standby working mode according to an embodiment of the present application;

[0033] FIG5 is a schematic diagram of a dual network card active / standby mode failover in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] In the description of this application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0036] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0037] 1 to 5 , a method for a TCP user-mode protocol stack to support dual network cards and diversified loads according to an optional embodiment of the present application includes:

[0038] Adding the SPDK transport layer and modifying the TCP user-mode protocol stack: Adding a new transport layer to SPDK to support the TCP protocol; modifying the TCP user-mode protocol stack and placing the network card packet processing function in the SPDK poller to support dual network cards;

[0039] Configuration and optimization: By sending data packets alternately through network cards, we can achieve linear growth in storage bandwidth and IOPS.

[0040] Exception handling and failover: When a network card fails to send packets for a long time, it is judged as an exception and a callback function is triggered after the timeout to resend the IO request to another available network card;

[0041] Monitoring and maintenance: Monitor network status, including the network card's packet sending and receiving status and network bandwidth utilization; regularly maintain and optimize the system to ensure system stability and performance.

[0042] It should be noted that by adding a new transport layer to SPDK to support the TCP protocol, modifying the TCP user-mode protocol stack, and placing the network card packet processing function in the SPDK poller, dual network cards can be supported, improving the flexibility and availability of the system.

[0043] Through configuration optimization, such as sending data packets alternately through network card sequencing, linear growth of storage bandwidth and IOPS can be achieved, improving system performance and efficiency.

[0044] Through the exception handling and fault switching mechanism, when a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to re-send the IO request to another available network card, enhancing the stability and reliability of the system.

[0045] Through the monitoring and maintenance mechanism, the network status can be monitored in real time, including the packet sending and receiving status of the network card and the network bandwidth utilization rate, and the system can be maintained and optimized regularly to ensure the stability and performance of the system, thereby improving the maintainability and manageability of the system.

[0046] In some embodiments, when there is a need to autonomously select a network card for IO processing, one of the dual network cards is selected as the primary network card and the other as the backup network card. The primary network card sends and receives network data packets. When the primary network card fails, it switches to the backup network card to send data packets.

[0047] In some embodiments, when a network card fails to send packets for a long time, it is determined to be abnormal.

[0048] The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

[0049] In some embodiments, when a network card fails to send packets for a long time, it is determined to be abnormal.

[0050] IO will attempt to be sent from the non-abnormal network card.

[0051] It should be noted that when it is necessary to independently select a network card for IO processing, a master-slave network card switching mechanism is adopted, which can automatically select a network card as the master network card to send and receive network data packets. When the master network card fails, it can automatically switch to the backup network card to send data packets. This mechanism improves the availability and adaptability of the system.

[0052] When a network card fails to send packets for a long time, it is judged as abnormal and the callback function is called to trigger the re-issuance of IO, or try to send it from a normal network card. This processing method can handle abnormal situations in a timely manner, ensure the normal operation of the system and data transmission, and avoid problems such as system downtime and data loss caused by network card failure.

[0053] The processing method of the above embodiment can effectively improve the stability and reliability of the system, ensure data transmission and system availability, and also improve the maintainability and manageability of the system, making it easier for administrators to discover and handle problems in a timely manner.

[0054] An optional embodiment of the present application further provides a system for a TCP user-mode protocol stack supporting dual network cards and diversified loads, which is applied to the above method. The system includes: a storage cluster,

[0055] The storage cluster includes: storage services and several storage nodes. The storage services include: file services, object services and block services. The file services, object services and block services are respectively provided with a transport layer, and the file services, object services and block services are respectively electrically connected to the storage nodes.

[0056] The file server is electrically connected to the storage cluster and includes a TCP user-mode protocol stack and several Ethernet interfaces for implementing network connectivity and data transmission. The file server is electrically connected to the file service, object service, and block service, respectively. The multiple Ethernet interfaces operate simultaneously.

[0057] An optional embodiment of the present application also provides a device for a TCP user-mode protocol stack to support dual network cards and support diversified loads. When the device is running, it executes a method for a TCP user-mode protocol stack to support dual network cards and support diversified loads.

[0058] Some examples and measurement data of this application are listed below to illustrate the significant progress of this application:

[0059] This solution uses storage as an example. SPDK is a storage acceleration component that enables high-performance I / O forwarding. Currently, it supports POSIX, TCP, and RDMA. To enable SPDK to support the TCP user-mode protocol stack, it first needs to be modified by adding SPDK's transport layer to the new transport protocol. Secondly, the TCP user-mode protocol stack needs to be modified to support dual NICs. Since the native TCP user-mode only supports a single NIC, it needs to be modified to support dual NICs. Figure 1 shows the SPDK + TCP user-mode protocol stack architecture supporting dual NICs.

[0060] As you can understand, network transmission performance is improved. Due to SPDK's high-performance forwarding mechanism, adding SPDK's transport layer can significantly improve network transmission speed and efficiency. It also supports dual network cards. While native TCP user-mode only supports a single network card, this method achieves dual network card support by modifying the TCP user-mode protocol stack. This allows network devices or computers to use multiple network cards simultaneously for data transmission, improving transmission flexibility and efficiency. It also supports diverse loads. Due to SPDK's flexibility, it can handle a variety of load conditions, including high and low loads, making this method adaptable to various application scenarios. Since the TCP user-mode protocol stack itself requires consideration of network card packet reception, to achieve high performance, the NIC's RX packet processing function can be placed in an SPDK poller. Since SPDK operates in polling mode, this ensures timely RX responses, significantly improving packet processing efficiency. SPDK's DPDK component supports dual network card probe operations. This solution simply requires the TCP protocol stack to handle packet transmission and reception for multiple network cards.

[0061] The TCP protocol stack's processing of RX packets has been described above. The following section will focus on the TX processing and the introduction of diversified payloads.

[0062] 1. TX alternating processing mode.

[0063] This mode primarily sends data packets in an alternating order based on the network interface cards (NICs). Similar to Bonding Mode 4, it achieves linear growth in storage bandwidth and IOPS. For example, a single 25Gb / s NIC has a bandwidth of approximately 3GB / s and only 70W IOPS. However, dual NICs can achieve a linear performance increase with storage bandwidth reaching 6GB / s and IOPS reaching 139.5W.

[0064] This mode implements NIC activity detection, using successful NIC packet transmission to provide feedback on NIC health. If one NIC fails to transmit packets for a prolonged period, it will be considered abnormal. Upon a timeout, the TCP flow in the Retry List of this abnormal NIC will invoke a callback function to trigger the retransmission of the I / O request. At this point, only one NIC is available, so the I / O request will be retried through the other NIC. Figure 2 shows the NIC switching flow when one NIC fails, when supporting two NICs simultaneously.

[0065] Figure 3 shows the switchover process after a single network card fails. This mode can solve the problem of insufficient performance of a single network card while achieving high availability.

[0066] It's understandable that by alternating data packet transmission and leveraging the advantages of dual NICs, storage bandwidth and IOPS can be linearly increased. Compared to single NIC mode, dual NIC mode can better meet the performance requirements of high-load scenarios. The TX alternating processing mode implements NIC liveness detection, detecting the health of the NICs by sending data packets. When a NIC fails to send packets for a long period of time, it is identified as an anomaly, triggering the appropriate processing flow to ensure the stability and reliability of network communications. This mode can address the insufficient performance of a single NIC. When a single NIC fails, data transmission and system availability can be guaranteed by switching to the other NIC. This automatic switching mechanism improves system reliability without interrupting network connectivity. By alternating data packet transmission, the load between the two NICs is balanced, preventing overloading of a single NIC, thereby improving the performance and stability of the entire system.

[0067] 2. TX master / slave processing mode

[0068] This mode is designed to support users' need to independently select a network card for I / O processing. Users can select a primary network card to send and receive network packets based on their business needs. Figure 4 shows a diagram of the primary and backup network card modes.

[0069] When the network card fails, the service can sense it in a very short time, so when sending subsequent data, another network card will be automatically selected to send data packets, as shown in Figure 5.

[0070] As you can understand, users can independently select a primary network card (NIC) to send and receive network packets based on their business characteristics, improving flexibility. If the primary NIC fails, the service will be notified quickly, avoiding long waits and congestion, thereby improving system availability and reliability. If the primary NIC fails, the system automatically selects a backup NIC to deliver packets, eliminating the need for manual intervention and streamlining operations and management. By configuring the primary and backup NICs, the load can be balanced between the two NICs, preventing overloading of a single NIC and improving overall system performance and stability.

[0071] In summary, the embodiment of the present application provides a method for a TCP user-mode protocol stack to support dual network cards and support diversified loads. Its main implementation scenario is a cloud computing storage scenario. It not only achieves the purpose of increasing the bandwidth and performance of the network card, but also achieves linear growth in performance. It provides good support for the high concurrency and high reliability requirements of the network within the storage cluster, and uses a dual network card diversified load strategy to meet the vast majority of actual application scenarios.

[0072] The above description is only an implementation example of the present application, but it cannot be used to limit the scope of the present application. Any structural changes made based on the present application, as long as they do not lose the essence of the present application, should be deemed to fall within the scope of protection of the present application and be subject to restrictions.

[0073] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present application can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules or steps and are not considered to be improper limitations on the present application.

[0074] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0075] Thus far, the technical solutions of the present application have been described in conjunction with the optional embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

[0076] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A method for a TCP user-mode protocol stack to support dual network cards and diversified loads, characterized in that: include: Adding SPDK transport layer and modifying TCP user-mode protocol stack: Add a new transport layer to SPDK to support TCP protocol; modify the TCP user-mode protocol stack and put the network card packet receiving function into the SPDK poller to support dual network cards; Configuration and optimization: The network card is used to sort and send data packets alternately to achieve linear growth of storage bandwidth and IOPS; Exception handling and fault switching: When a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to resend the IO request to another available network card; Monitoring and maintenance: monitoring network status, including the network card's packet sending and receiving status and network bandwidth usage; Regularly maintain and optimize the system to ensure system stability and performance.

2. The method for supporting dual network cards and diversified loads by a TCP user mode protocol stack according to claim 1, characterized in that: When there is a need to independently select a network card for IO processing, one of the dual network cards is selected as the primary network card and the other as the backup network card. The primary network card sends and receives network data packets. When the primary network card fails, it switches to the backup network card to send data packets.

3. The method for supporting dual network cards and diversified loads by a TCP user mode protocol stack according to claim 1, characterized in that: When a network card fails to send packets for a long time, it is judged as abnormal. The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

4. The method for supporting dual network cards and diversified loads by a TCP user mode protocol stack according to claim 1, characterized in that: When a network card fails to send packets for a long time, it is judged as abnormal. IO will attempt to be sent from the non-abnormal NIC.

5. A system for supporting dual network cards and diversified loads with a TCP user-mode protocol stack, applied to the method for supporting dual network cards and diversified loads with a TCP user-mode protocol stack as described in any one of claims 1 to 4, characterized in that: The system comprises: a storage cluster, The storage cluster includes: a storage service and several storage nodes, the storage service includes: a file service, an object service and a block service, the file service, the object service and the block service are respectively provided with a transport layer, and the file service, the object service and the block service are respectively electrically connected to the storage nodes.

6. The system for supporting dual network cards and diversified loads according to claim 5, characterized in that: Also includes: A file server is electrically connected to the storage cluster, and the file server comprises: a TCP user mode protocol stack and a plurality of Ethernet interfaces for realizing network connection and data transmission functions.

7. The system for supporting dual network cards and diversified loads according to claim 6, characterized in that: The file server is electrically connected to the file service, the object service, and the block service, respectively.

8. The system for supporting dual network cards and diversified loads according to claim 6, characterized in that: Several of the Ethernet interfaces operate simultaneously.

9. A device for supporting dual network cards and diversified loads with a TCP user mode protocol stack, characterized in that: When it is running, the method for the TCP user mode protocol stack to support dual network cards and diversified loads as described in any one of claims 1 to 4 is executed.

10. An application of a method for a TCP user-mode protocol stack supporting dual network cards and supporting diversified loads as described in any one of claims 1-4 in cloud host virtualization scenarios, container scenarios, GPU AI scenarios, and distributed storage scenarios.

Images