A method, system and device for TCP user-mode protocol stack to support dual network cards and diversified loads

Abstract

The present invention relates to the field of system communication technology, and specifically to a method, system and device for a TCP user-mode protocol stack to support dual network cards and diversified loads, including: SPDK transport layer addition and TCP user-mode protocol stack modification: adding a new transport layer to SPDK to support TCP protocol; modifying the TCP user-mode protocol stack, placing the network card packet receiving processing function into the poller of SPDK, for supporting dual network cards; configuration and optimization: sending data packets alternately through network card sorting, for achieving linear growth of storage bandwidth and IOPS; exception handling and fault switching: when a network card fails to send packets for a long time, it is judged as an exception, and after timeout, a callback function is triggered to re-send IO requests to another available network card; and monitoring and maintenance. The user-mode protocol stack dual network card performance of the present invention can be increased linearly, and the performance can achieve linear growth.

Description

Technical Field

[0001] The present invention relates to the technical field of system communication, and in particular to a method for a TCP user mode protocol stack to support dual network cards and diversified loads. Background Art

[0002] Computing, storage and network are three important components of cloud computing. With the continuous improvement of CPU computing power, network performance has also improved, and the demand for high performance storage has become more and more urgent. Many old models have multiple 25Gb / s network cards. With the continuous iteration of models, there will be more and more old models. In order to solve the bottleneck of hardware, the NVMe protocol came into being. However, in cloud computing, high-performance cloud disks are a demand with a large demand and wide coverage. Only NVMe hard disks cannot meet the actual user needs.

[0003] In order to solve the problem of low network transmission performance, NVMe over RDMA is a good choice. It mainly uses the high performance and zero-copy technology of RDMA to achieve high-performance IO transmission and ultra-high cloud disk performance. However, RDMA has its own technical limitations. That is, if cross-AZ access is performed in cloud computing, the network performance will become smaller and cannot meet the actual production environment. The TCP protocol is suitable for storage transmission in the kernel protocol stack, but its performance is poor.

[0004] Therefore, cloud computing storage scenarios urgently need a high-performance network transmission protocol with good compatibility. Summary of the invention

[0005] In order to achieve the above objectives, the present invention provides a method for a TCP user-mode protocol stack to support dual network cards and support diversified loads, so as to solve the problem that it is difficult to achieve low latency, high concurrency and high performance when accelerating the network in scenarios such as storage clusters, virtualization, GPU AI, etc.

[0006] The present invention proposes a method for a TCP user-mode protocol stack to support dual network cards and diversified loads, comprising:

[0007] Adding SPDK transport layer and modifying TCP user-mode protocol stack: Add a new transport layer to SPDK to support TCP protocol; modify the TCP user-mode protocol stack and put the network card packet receiving function into the SPDK poller to support dual network cards;

[0008] Configuration and optimization: The network card is used to sort and send data packets alternately to achieve linear growth of storage bandwidth and IOPS;

[0009] Exception handling and fault switching: When a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to resend the IO request to another available network card;

[0010] Monitoring and maintenance: monitoring the network status, including the packet sending and receiving status of the network card and the network bandwidth utilization rate; regularly maintaining and optimizing the system to ensure the stability and performance of the system.

[0011] Furthermore, when there is a need to independently select a network card for IO processing, one of the dual network cards is selected as the main network card and the other as the backup network card. The main network card sends and receives network data packets. When the main network card fails, it switches to the backup network card to send data packets.

[0012] Furthermore, when a network card fails to send packets for a long time, it is judged as abnormal.

[0013] The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

[0014] Furthermore, when a certain network card fails to send packets for a long time, it will be judged as abnormal, and the IO will try to be sent from a normal network card.

[0015] The embodiment of the present invention provides a data-driven front-end resource loading optimization method, which has the following beneficial effects compared with the prior art:

[0016] The user-mode protocol stack of the present invention is currently the only protocol stack that supports dual network cards, and the performance of the network card increases linearly; in the actual production environment, the bandwidth of a single 25Gb / s single network card is about 3GB / s, and the IOPS is only 70W IOPS. The storage bandwidth of the 2x25Gb / s mode can reach 6GB / s, and the IOPS can reach 139.5W. Experiments have proved that the performance in the dual network card mode achieves linear growth; the TCP user-mode protocol stack is not only close to the RDMA performance, but also supports KernelTCP while supporting itself, and has very good compatibility, which can well meet the needs of network transmission in cloud computing; in the cloud computing scenario, there are many old models with multiple 25Gb / s network cards. With the continuous iteration of models, there will be more and more old models. In order to make good use of these models and reduce enterprise costs, the TCP user-mode protocol stack needs to support multiple network cards to achieve linear growth in performance, so that these old models can play a greater value and realize resource reuse.

[0017] On the other hand, the present invention proposes a system for TCP user-mode protocol stack supporting dual network cards and supporting diversified loads, which is applied to the method for TCP user-mode protocol stack supporting dual network cards and supporting diversified loads. The system includes: a storage cluster,

[0018] The storage cluster includes: a storage service and a plurality of storage nodes, the storage service includes: a file service, an object service and a block service, the file service, the object service and the block service are respectively provided with a transport layer, and the file service, the object service and the block service are respectively electrically connected to the storage nodes.

[0019] Furthermore, it also includes: a file server, which is electrically connected to the storage cluster, and the file server includes: a TCP user mode protocol stack and several Ethernet interfaces for realizing network connection and data transmission functions.

[0020] Furthermore, the file server is electrically connected to the file service, the object service and the block service respectively.

[0021] Furthermore, several of the Ethernet interfaces work simultaneously.

[0022] On the other hand, the present invention also proposes a device for a TCP user-mode protocol stack to support dual network cards and to support diversified loads. When the device is running, the method for a TCP user-mode protocol stack to support dual network cards and to support diversified loads is executed.

[0023] On the other hand, the present invention also provides a method for TCP user-mode protocol stack to support dual network cards and diversified loads, and its application in cloud host virtualization scenarios, container scenarios, GPU AI scenarios and distributed storage scenarios.

[0024] It can be understood that the above-mentioned method, system and device for supporting dual network cards and diversified loads of the TCP user mode protocol stack have the same beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of a storage scenario supported by SPDK and TCP user-mode protocol stack in an embodiment of the present invention;

[0027] Figure 2 This is an IO flow chart before a failure occurs in a dual network card in an embodiment of the present invention;

[0028] Figure 3 is a switching flow chart after a single network card fails in an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of dual network card active / standby working mode in an embodiment of the present invention;

[0030] Figure 5 Schematic diagram of dual network card active / standby mode fault switching in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0033] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0034] See also Figure 1-5 As shown, a method for a TCP user mode protocol stack to support dual network cards and diversified loads in a preferred embodiment of the present invention includes:

[0035] Adding SPDK transport layer and modifying TCP user-mode protocol stack: Add a new transport layer to SPDK to support TCP protocol; modify the TCP user-mode protocol stack and put the network card packet receiving function into SPDK's poller to support dual network cards;

[0036] Configuration and optimization: The network card is used to sort and send data packets alternately to achieve linear growth of storage bandwidth and IOPS;

[0037] Exception handling and fault switching: When a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to resend the IO request to another available network card;

[0038] Monitoring and maintenance: Monitor network status, including the network card's packet sending and receiving status and network bandwidth utilization; regularly maintain and optimize the system to ensure system stability and performance.

[0039] It should be noted that by adding a new transport layer in SPDK to support the TCP protocol, modifying the TCP user-mode protocol stack, and placing the network card packet receiving processing function in the SPDK poller, it is possible to support dual network cards and improve the flexibility and availability of the system.

[0040] Through configuration optimization, such as sending data packets alternately through network card sequencing, linear growth of storage bandwidth and IOPS can be achieved, improving system performance and efficiency.

[0041] Through the exception handling and fault switching mechanism, when a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to re-send the IO request to another available network card, thereby enhancing the stability and reliability of the system.

[0042] Through the monitoring and maintenance mechanism, the network status can be monitored in real time, including the packet receiving and sending status of the network card and the network bandwidth utilization rate, and the system can be maintained and optimized regularly to ensure the stability and performance of the system, thereby improving the maintainability and manageability of the system.

[0043] In some of the embodiments, when there is a need to autonomously select a network card for IO processing, one of the dual network cards is selected as the primary network card and the other is selected as the backup network card. The primary network card sends and receives network data packets. When the primary network card fails, it switches to the backup network card to send data packets.

[0044] In some embodiments, when a network card fails to send packets for a long time, it is determined to be abnormal.

[0045] The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

[0046] In some embodiments, when a network card fails to send packets for a long time, it is determined to be abnormal.

[0047] IO will attempt to be sent from the non-abnormal NIC.

[0048] It should be noted that when it is necessary to independently select a network card for IO processing, a switching mechanism between the primary and backup network cards is adopted, which can automatically select a network card as the primary network card to send and receive network data packets. When the primary network card fails, it can automatically switch to the backup network card to send data packets. This mechanism improves the availability and adaptability of the system.

[0049] When a network card fails to send packets for a long time, it is judged as abnormal and the callback function is called to trigger the re-issuance of IO, or try to send it from a normal network card. This processing method can handle abnormal situations in time to ensure the normal operation of the system and data transmission, while avoiding problems such as system downtime and data loss caused by network card failure.

[0050] The processing method of the above embodiment can effectively improve the stability and reliability of the system, ensure data transmission and system availability, and also improve the maintainability and manageability of the system, making it easier for administrators to discover and handle problems in a timely manner.

[0051] The preferred embodiment of the present invention also provides a system for supporting dual network cards and diversified loads with a TCP user-mode protocol stack, which is applied to the above method and includes: a storage cluster,

[0052] The storage cluster includes: a storage service and several storage nodes. The storage service includes: a file service, an object service and a block service. The file service, the object service and the block service are respectively provided with a transport layer, and the file service, the object service and the block service are respectively electrically connected to the storage nodes.

[0053] The file server is electrically connected to the storage cluster. The file server includes: a TCP user-mode protocol stack and several Ethernet interfaces for realizing network connection and data transmission functions. The file server is electrically connected to the file service, object service and block service respectively. Several Ethernet interfaces work simultaneously.

[0054] The preferred embodiment of the present invention also provides a device for a TCP user-mode protocol stack to support dual network cards and support diversified loads. When the device is running, it executes a method for a TCP user-mode protocol stack to support dual network cards and support diversified loads.

[0055] Some embodiments and measurement data of the present invention are listed below to illustrate the significant progress of the present invention:

[0056] This solution takes the storage scenario as an example. SPDK is a component of storage acceleration, which can achieve high-performance forwarding in IO. Currently, it supports POSIX, TCP, and RDMA network transmission protocols. In order for SPDK to support the TCP user-state protocol stack, it is first necessary to modify it and add the SPDK transport layer to the new transmission protocol; secondly, the TCP user-state protocol stack must be modified to support dual network cards. Since the native TCP user state only supports single network card mode, it needs to be modified if it supports dual network cards. SPDK+TCP user-state protocol stack supports dual network card architecture such as Figure 1 shown.

[0057] It can be understood that the performance of network transmission is improved. Due to the high-performance forwarding mechanism of SPDK, adding the transport layer of SPDK can greatly improve the speed and efficiency of network transmission. Support dual network cards. The native TCP user state only supports the single network card mode, and this method realizes the support of dual network cards by modifying the TCP user state protocol stack, which enables network devices or computers to use multiple network cards for data transmission at the same time, improving the flexibility and efficiency of transmission. Support diverse loads. Due to the flexibility of SPDK, it can cope with various load conditions, including high load and low load, which makes this method adaptable to different application scenarios. Since the TCP user state protocol stack requires the protocol itself to consider the network card packet reception, in order to achieve high-performance packet reception, the packet processing function of the RX direction of the network card can be placed in a poller of SPDK. Since SPDK works in polling mode, RX will get a timely response, greatly improving the packet processing efficiency. The DPDK component of SPDK supports dual network card Probe operation. This solution only needs to let the TCP protocol stack process the packet receiving and sending behaviors of multiple network cards.

[0058] The TCP protocol stack's processing of RX direction data packets has been introduced above. The following will focus on the TX processing method and the introduction of diversified loads.

[0059] 1. TX alternating processing mode.

[0060] This mode mainly sends data packets alternately according to the order of the network cards, which is the same as the Bonding Mode 4 mode, and can achieve linear growth of storage bandwidth and IOPS. For example, the bandwidth of a 25Gb / s single network card is about 3GB / s, and the IOPS is only 70W IOPS. If it is a dual network card mode, the storage bandwidth can reach 6GB / s, and the IOPS can reach 139.5W, and the performance can achieve linear growth.

[0061] This mode implements NIC detection and feedback of NIC health status through successful NIC packet sending. When one of the NICs fails to send packets for a long time, it will be judged as abnormal. The TCP Flow in the Retry List of this abnormal NIC will call the callback function to trigger the re-issuance of this IO when the timeout occurs. At this time, there will be only one available NIC, so the IO will try to be sent from another NIC. Figure 2 As shown in FIG. 1 , it is a flowchart of network card switching when one network card fails when two network cards are supported at the same time.

[0062] like Figure 3 The figure shows the switching process after a single network card fails. This mode can solve the problem of insufficient performance of a single network card and achieve high availability.

[0063] It is understandable that by sending data packets alternately and taking advantage of the dual network cards, storage bandwidth and IOPS can be increased linearly. Compared with the single network card mode, the dual network card mode can better meet the performance requirements in high-load scenarios. The TX alternating processing mode implements the network card detection function, which detects the health of the network card by sending data packets. When a network card fails to send packets for a long time, it will be judged as abnormal, triggering the corresponding processing flow to ensure the stability and reliability of network communication. This mode can solve the problem of insufficient performance of a single network card. When a single network card fails, it can switch to another network card to ensure data transmission and system availability. This automatic switching mechanism can improve the reliability of the system without interrupting the network connection. By sending data packets alternately, the load of the two network cards can be balanced to avoid overloading of a single network card, thereby improving the performance and stability of the entire system.

[0064] 2. TX master-slave processing mode

[0065] This mode is mainly to support the user's need to select a network card to perform IO processing. Users can select a primary network card to send and receive network data packets based on the characteristics of their own business. Figure 4 The figure shows the master / slave mode of the network card.

[0066] When the network card fails, the service can sense it in a very short time, so when sending subsequent data, it will automatically select another network card to send data packets. Figure 5 shown.

[0067] It is understandable that users can choose a primary network card to send and receive network data packets according to the characteristics of their own business, which improves flexibility. When the primary network card fails, the business can sense it in a very short time, avoiding long waiting and blocking, and improving the availability and reliability of the system. When the primary network card fails, the system will automatically select the backup network card to send data packets without manual intervention, simplifying the operation and management process. By setting the primary and backup network cards, the load of the two network cards can be balanced to avoid overloading of a single network card, thereby improving the performance and stability of the entire system.

[0068] In summary, the embodiment of the present invention provides a method for a TCP user-mode protocol stack to support dual network cards and support diversified loads. Its main implementation scenario is a cloud computing storage scenario. It not only achieves the purpose of increasing the bandwidth and performance of the network card, but also achieves linear growth in performance. It provides good support for the requirements of high concurrency and high reliability of the network within the storage cluster, and uses a dual network card diversified load strategy to meet the vast majority of actual application scenarios.

[0069] The above is only an example of implementation of the present invention, but it cannot be used to limit the scope of the present invention. Any structural changes made according to the present invention should be regarded as falling within the protection scope of the present invention and subject to restrictions as long as they do not lose the essence of the present invention.

[0070] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not regarded as improper limitations of the present invention.

[0071] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0072] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for a TCP user-mode protocol stack to support dual network cards and diversified loads, characterized in that: include: Adding SPDK transport layer and modifying TCP user-mode protocol stack: Add a new transport layer to SPDK to support TCP protocol; modify the TCP user-mode protocol stack and put the network card packet receiving function into the SPDK poller to support dual network cards; Configuration and optimization: The network card is used to sort and send data packets alternately to achieve linear growth of storage bandwidth and IOPS; Exception handling and fault switching: When a network card fails to send packets for a long time, it is judged as an exception, and after the timeout, the callback function is triggered to resend the IO request to another available network card; Monitoring and maintenance: monitoring network status, including the packet sending and receiving status of the network card and the network bandwidth utilization rate; Regularly maintain and optimize the system to ensure system stability and performance; When there is a need to select a network card for IO processing, one of the dual network cards is selected as the primary network card and the other as the backup network card. The primary network card sends and receives network data packets. When the primary network card fails, it switches to the backup network card to send data packets. When a network card fails to send packets for a long time, it is judged as abnormal. The TCP Flow in the Retry List of the abnormal network card will trigger the re-issuance of IO by calling the callback function when it times out.

2. The method for supporting dual network cards and diversified loads by a TCP user mode protocol stack according to claim 1, characterized in that: When a network card fails to send packets for a long time, it is judged as abnormal. IO will attempt to be sent from the non-abnormal NIC.

3. A system for supporting dual network cards and diversified loads with a TCP user-mode protocol stack, applied to the method for supporting dual network cards and diversified loads with a TCP user-mode protocol stack as described in any one of claims 1 to 2, characterized in that: include: Storage cluster, The storage cluster includes: a storage service and a plurality of storage nodes, the storage service includes: a file service, an object service and a block service, the file service, the object service and the block service are respectively provided with a transport layer, and the file service, the object service and the block service are respectively electrically connected to the storage nodes.

4. The system for supporting dual network cards and diversified loads according to claim 3, characterized in that: Also includes: A file server is electrically connected to the storage cluster, and the file server comprises: a TCP user mode protocol stack and a plurality of Ethernet interfaces for realizing network connection and data transmission functions.

5. The system for supporting dual network cards and diversified loads according to claim 4, characterized in that: The file server is electrically connected to the file service, the object service, and the block service, respectively.

6. The system for supporting dual network cards and diversified loads according to claim 5, characterized in that: Several of the Ethernet interfaces operate simultaneously.

7. An application of a method for a TCP user-mode protocol stack supporting dual network cards and supporting diversified loads as described in any one of claims 1-2 in a cloud host virtualization scenario, a container scenario, a GPUAI scenario, or a distributed storage scenario.

Images