A Multi-Path Selection Method for Data Channels Oriented to Distributed Storage

Abstract

The present invention relates to the field of network communication technologies, and particularly to a multi-path selection method for data channels oriented to distributed storage, including the following steps: S1: A first node sends a probe packet with a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network interface, and a receiving network interface according to the network connection mode, and a second node generates a reply probe packet after receiving it; S2: The first node obtains path information including the source port, the sending network interface, and the receiving network interface according to the reply probe packet, and selects a path; S3: The first node obtains the path selection result, and establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes a multi-path data channel, and periodically performs path health checks. The present invention realizes TCP / RDMA path detection, uses multiple random source ports for detection, and ensures that the network interfaces used by each path are different, and can fully utilize the bandwidth between multiple network interfaces.

Description

Technical Field

[0001] The present invention relates to the field of network communication technologies, and in particular, to a multi-path selection method for data channels for distributed storage and a storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data shows an explosive growth trend. The traditional single-machine storage method has been difficult to meet the needs of storing, managing, and efficiently accessing massive data, and a distributed storage system has emerged as the times require. It disperses data storage on multiple physical nodes connected by a network and provides unified data storage services through collaborative work, with many advantages such as scalability, high reliability, and high performance. In a distributed storage system, multiple physical nodes in a cluster rely on the network for communication, and the data channel is responsible for data transmission between nodes through the network.

[0003] Currently, the data channel between two nodes usually depends on a single network connection, that is, a single path. The common connection methods of the path mainly include two types: TCP and RDMA. However, the data channel of a single path is limited by the bandwidth of a single network interface in terms of performance, and the failure of a single network interface will cause the data transmission to be interrupted, thereby affecting the availability and overall performance of the cluster. Summary of the Invention

[0004] The purpose of the present invention is to solve the drawbacks existing in the prior art, and provides a multi-path selection method for data channels for distributed storage, including the following steps:

[0005] S1: The first node sends a probe packet with a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network interface, and a receiving network interface according to the network connection method, and the second node receives the probe packet and generates a reply probe packet;

[0006] S2: The first node obtains path information including the source port and network interface information according to the received reply probe packet, and selects a path according to the sending network interface and the receiving network interface in the network interface information;

[0007] S3: After the first node obtains the path selection result, it establishes corresponding network TCP / RDMA connections according to the source port / RDMA flow label of the selected multiple paths, that is, establishes a multi-path data channel, and regularly performs path health checks.

[0008] Preferably, in step S1, the first node sends a probe packet according to the network connection method, which further includes:

[0009] If the network connection mode is a TCP connection, a TCP probe packet is sent through a raw socket, and the TCP probe packet includes a TCP header and a TCP probe packet body;

[0010] If the network connection mode is an RDMA connection, an RDMA probe packet is sent through an unreliable datagram mode UD, and the RDMA probe packet includes routing information and the RDMA probe packet body.

[0011] Preferably, sending a TCP probe packet through a raw socket further includes:

[0012] The first node specifies the second node IP and the second node port through the sendto function of the raw socket, and fills in the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header, and the Ethernet header;

[0013] The first node generates a plurality of the TCP probe packet bodies for each network interface according to different source ports, and sends the TCP probe packet according to the TCP header and the TCP probe packet body.

[0014] Preferably, the second node receives the TCP probe packet and generates a TCP reply probe packet, further including:

[0015] The second node starts a packet capture tool to capture the TCP probe packet, and identifies the version number through a filter. If the identification result meets the requirements, the TCP probe packet is received. If the identification result does not meet the requirements, the TCP probe packet is discarded;

[0016] The second node obtains the second node receiving network interface used by the current path according to the captured TCP probe packet, fills it into the receiving network interface to construct a first TCP reply probe packet, and sends the first TCP reply probe packet through the raw socket;

[0017] The second node obtains the second node sending network interface used by the current path according to the captured TCP probe packet, fills it into the sending network interface to construct a second TCP reply probe packet, and sends the second TCP reply probe packet through the raw socket.

[0018] Preferably, sending an RDMA probe packet through an unreliable datagram mode UD further includes:

[0019] The first node and the second node start an RDMA UD queue pair QP on each network interface, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information of each of the RDMA UD queue pair QPs on the second node, including the local identifier LID, the queue pair number QPN, and the global identifier GID;

[0020] The first node randomly generates a number of the RDMA flow labels for each network interface, and fills the RDMA flow labels into the source port of the RDMA probe packet body to generate an RDMA probe packet body;

[0021] The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends the RDMA probe packet including the second node routing information and the RDMA probe packet body to the second node through the RDMA work request WR.

[0022] Preferably, the second node receives the RDMA probe packet and generates an RDMA reply probe packet, further including:

[0023] Only one UD queue pair QP in the RDMA UD queue pair QP of the second node can receive the RDMA probe packet, and the second node receiving network interface is recorded according to the RDMA probe packet;

[0024] The second node fills the second node receiving network interface into the receiving network interface of the RDMA reply probe packet body, and sends the RDMA reply probe packet to the first node by sending an RDMA work request WR.

[0025] Preferably, in step S2, the first node obtains path information according to the received reply probe packet, further including:

[0026] If the network connection mode is a TCP connection, after the first node receives the two reply probe packets, namely the first TCP reply probe packet and the second TCP reply probe packet, the first node can perceive the source port used by each path of the TCP probe packet, the first node sending network interface, the first node receiving network interface, the second node sending network interface, and the second node receiving network interface;

[0027] If the network connection mode is RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply probe packet, it records the source port for receiving the RDMA reply probe packet. For an RDMA connection, the sending network interface and the receiving network interface of a connection are the same. Therefore, the sending network interface of the source end is the same as the receiving network interface of the source end, and the sending network interface of the peer end is the same as the receiving network interface of the peer end. At this time, the first node can sense the sending network interface of the first node, the receiving network interface of the first node, the sending network interface of the second node, and the receiving network interface of the second node for each path.

[0028] Preferably, in step S3, the path health check is performed regularly, and further includes:

[0029] The path health check is performed within a set time, and the sending network interfaces of all current valid paths are used to mark the locally used network interfaces, so as to sense whether there is an unused network interface currently; if there is, it will trigger the failure reconnection of multiple paths to perform new path detection, and determine whether the newly detected path uses a different network interface from the existing valid paths. If there is a path that meets the conditions, a new path is established using the source port / the RDMA flow label. If there is no path that meets the conditions, the process ends.

[0030] Preferably, the continuous reconnection times that trigger the failure reconnection but do not actually complete the paths are recorded. If this number exceeds the set number, the set time is extended to avoid frequent triggering of reconnection. And as long as a disconnection occurs on one path, the set time will be reset to the initial set time, and the continuous reconnection times will be set to 0, so as to quickly complete all paths.

[0031] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the data channel multi-path selection method for distributed storage as described in any one of the embodiments.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] The present invention realizes TCP / RDMA path detection by the first node sending a probe packet with a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network interface, and a receiving network interface according to the network connection mode, and the second node receiving the probe packet and generating a reply probe packet.

[0034] The first node of the present invention obtains network interface information including the source port, the sending network interface, and the receiving network interface based on the received reply probe packet, and selects a path according to the network interface information, so as to realize probing by using multiple random source ports. The first node can obtain all network interfaces used by multiple paths and ensure that the network interfaces used by each path are different.

[0035] The present invention establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes a multi-path data channel, and regularly performs path health checks to realize the use of all current valid paths, so as to sense whether there are unused network interfaces currently, and supplement new paths to fully utilize the bandwidth between multiple network interfaces. Brief Description of the Drawings

[0036] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.

[0037] Figure 1 It is a flowchart of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0038] Figure 2 It is a TCP probe flowchart of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0039] Figure 3 It is an RDMA probe flowchart of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0040] Figure 4 It is a data channel establishment flowchart of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0041] Figure 5 It is an example diagram of data channel establishment of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0042] Figure 6 It is a path check flowchart of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0043] Figure 7 It is an example diagram of path check of a multi-path selection method for a data channel for distributed storage according to the present invention;

[0044] Figure 8 It is an example diagram of a scenario with frequent reconnection triggers of a multi-path selection method for a data channel for distributed storage according to the present invention. Detailed Embodiments

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the scope of protection of the present application.

[0046] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", and "the" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0047] Embodiment 1

[0048] To solve the problem of poor performance and insufficient fault tolerance of a single network interface under a TCP connection, the current solutions usually improve the bandwidth and reliability of network connections through link aggregation (Bonding) technology. Link aggregation technology can combine multiple network interfaces into a logical link, which can ensure that even if a certain network interface fails, other network interfaces can still bear the traffic, thus avoiding network interruption and enhancing the availability of the system. Among them, the common modes of link aggregation include active-backup, balance-slb, and balance-tcp, etc. Although link aggregation can theoretically provide the bandwidth of multiple network interfaces, in actual use, the active-backup mode only supports primary-backup switching and cannot use the bandwidth of multiple network interfaces simultaneously; although the balance-slb and balance-tcp modes support load balancing, the path selection depends on the hash result, and most data streams may be assigned to some network interfaces, unable to ensure the uniform distribution of data streams. Especially when the number of connections is small, it is easy to occur that some network interfaces are overloaded while other network interfaces are idle. This load imbalance problem limits the bandwidth utilization rate of multiple network interfaces and affects the overall network performance.

[0049] Taking Ceph as an example, when using TCP connections as data channels in a multi-network interface environment, link aggregation technology is usually relied on to improve bandwidth and availability. For example, when using the balance-tcp mode, the distribution of traffic depends on hash algorithms (such as source IP, destination IP, port, etc.). To fully utilize the bandwidth of all network interfaces, multiple connections usually need to be established. However, due to the certain randomness of hash-based load balancing, it cannot be guaranteed that the bandwidth of each network interface is fully utilized, or it is difficult to ensure the load balancing effect between network interfaces. This unbalanced load distribution will affect the overall performance of the system, especially in high-concurrency read-write scenarios, where the bandwidth cannot be maximally utilized.

[0050] Compared with TCP connections, RDMA connections can provide lower latency and higher throughput. However, in a multi-network interface environment, existing data channels using RDMA connections still have problems of insufficient bandwidth utilization and insufficient fault tolerance. Many existing data channels using RDMA connections only support communication through a single network interface. For example, Nutanix only supports establishing an RDMA connection through a single network interface. If this network interface fails, the data channel degrades to communicate using a TCP connection, resulting in a significant decline in data transmission performance and a substantial reduction in throughput, unable to meet the requirements of high reliability and high performance. In addition, this solution cannot achieve bandwidth aggregation in a multi-network interface environment and cannot fully utilize the performance advantages when combining a multi-network interface environment with RDMA.

[0051] In vSAN, although the data channel using an RDMA connection supports network interface card failover (NIC failover), it also lacks support for more complex network interface card teaming such as the Link Aggregation Control Protocol (LACP) or IP-hash. When a network interface card fails, the system will automatically switch to the standby network interface card to continue communication, thus maintaining the stability of data transmission. However, this solution also cannot effectively utilize the bandwidth of multiple network interfaces.

[0052] Therefore, although RDMA offers the advantages of low latency and high bandwidth, in a multi-network interface environment, the existing design of data channels using RDMA connections fails to effectively solve the problems of bandwidth aggregation and fault tolerance, resulting in the underutilization of RDMA's performance. The limitations of insufficient bandwidth and fault tolerance make the existing RDMA data channel solutions unable to meet the requirements of distributed storage systems in terms of high availability and high performance, unable to fully utilize the low-latency and high-bandwidth advantages of RDMA in a multi-network interface environment, and restricting the application potential of the system in high-availability and high-performance scenarios.

[0053] To solve the above problems, the present embodiment has the following preconditions:

[0054] 1. Bond type

[0055] Currently, both Linux Bond and Open vSwitch (OvS) Bond on the Linux system support network link aggregation, which can combine multiple network interfaces into a single logical link, thereby increasing bandwidth and providing redundancy. However, Linux Bond has certain limitations in supporting RDMA devices. Specifically, in the Linux Bond RDMA mode, only multiple network interfaces of the same network card can be aggregated, and port aggregation across different network cards is not possible, nor can it provide fault tolerance between network cards. After adopting Linux Bond, the upper-layer application software can only see a virtual network device after aggregation and can no longer specify in more detail which network card the data is sent / received through. Therefore, Linux Bond cannot meet the requirements of this embodiment, and this embodiment requires the use of OvS Bond for network link aggregation and the aggregation of ports between different network cards.

[0056] 2. Bond mode

[0057] Ovs Bond mainly has several modes, namely active-backup, balance-slb, and balance-tcp. Among them, the active-backup mode only uses the active slave to send network packets and cannot utilize the bandwidth of multiple network interfaces; the balance-slb mode is hashed based on the source MAC. Considering that each node in the distributed storage system usually uses the MAC address of the Bond for communication after configuring the Bond, this mode also cannot utilize multiple network interfaces; the balance-tcp mode is hashed based on the source and destination IPs, source and destination TCP / UDP ports, and selects the network interface for sending network packets according to the hashing result. Therefore, as long as the data channel can reasonably establish multiple TCP / RDMA connections, all network interfaces in the Bond can be used for sending.

[0058] 3. Switch configuration

[0059] When using the Ovs Bond and balance-tcp modes, multiple connections established within the data channel can utilize multiple network interfaces of the sender, but the receiving network interfaces used by the receiver are controlled by the port aggregation (Port Channel) of the switch. The port aggregation of the switch can generally configure the fields used for hashing. If these fields of multiple connections are different, there is a chance to hash to different network interfaces of the receiver. This embodiment requires that the port aggregation of the switch be configured to perform hashing based on source-destination-port.

[0060] Please refer to Figure 1 As shown, the multi-path selection method for the data channel for distributed storage provided in this embodiment ensures that in a multi-network interface environment, the data channel from the first node (client) to the second node (server) in the distributed storage system can transmit data through multiple TCP or RDMA connections (i.e., multi-path). These connection combinations will make full use of all network interfaces and link capabilities of the client and the server. The data channel can provide a mode of RDMA cross-network card aggregation to provide fault tolerance for network card failures and support the flexible control of load balancing of data traffic by upper-layer applications, including the following steps:

[0061] S1: The first node sends a probe packet with a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network interface, and a receiving network interface according to the network connection method. The second node receives the probe packet and generates a reply probe packet. Specifically, in this embodiment, the format of the probe packet body is Version|Type|Message ID|Src Port|Send Iface|Recv Iface|Reserve. Both TCP and RDMA use the probe packet body in the same format, and this embodiment does not limit the format and can be changed according to needs.

[0062] More preferably, the meanings of the respective fields of the probe packet body are as follows:

[0063] Version: Version number, used to distinguish from other TCP packets and support subsequent upgrades;

[0064] Type: Probe packet type, indicating whether it is a request or a reply;

[0065] Message ID: An identifier of a request probe packet. The reply probe packet uses the same ID as this probe packet;

[0066] Src Port: The source TCP port used by this probe packet;

[0067] Send Iface: Used for the reply probe packet to transfer the sending network interface of the peer;

[0068] Recv Iface: Used for replying to probe packets and used to transfer the peer receiving network interface;

[0069] Reserve: Reserved field.

[0070] Please refer to Figure 2 and Figure 3 As shown, in step S1, the first node sends a probe packet according to the network connection mode, which further includes:

[0071] If the network connection mode is a TCP connection, the first node needs to start a packet capture tool for capturing and replying to probe packets before sending the probe packet. After that, the TCP probe packet is sent through the raw socket Raw Socket. The TCP probe packet includes a TCP header and a TCP probe packet body;

[0072] If the network connection mode is an RDMA connection, the RDMA probe packet is sent through the unreliable datagram mode UD. The RDMA probe packet includes routing information and an RDMA probe packet body. Specifically, in this embodiment, the RDMA probe packet cannot use the raw socket to send the probe packet and use the packet capture tool for reception. The reason is that the packet capture tool needs the kernel to participate in capturing the RDMA probe packet, which will cause a large performance overhead, and currently the packet capture tool does not support non-blocking capture of the RDMA probe packet.

[0073] Preferably, sending the TCP probe packet through the raw socket Raw Socket further includes:

[0074] The first node specifies the second node IP and the second node port through the sendto function of the raw socket Raw Socket, and fills the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header and the Ethernet header;

[0075] The first node generates several TCP probe packet bodies for each network interface according to different source ports, and sends the TCP probe packet according to the TCP header and the TCP probe packet body. Specifically, in this embodiment, 10 TCP probe packets are sent, and the source ports of the TCP probe packets are different, and all network interfaces used by each path are detected, and the probe packets are captured through the packet capture tool to obtain the first node sending network interface used by the current path, that is, the source end sending network interface;

[0076] Preferably, the second node receives the TCP probe packet and generates a TCP reply probe packet, which further includes:

[0077] The second node starts the packet capture tool to capture TCP probe packets, and uses a filter to identify the version number. If the identification result meets the requirements, the TCP probe packet is received; if the identification result does not meet the requirements, the TCP probe packet is discarded.

[0078] The second node obtains the second node receiving network interface used by the current path based on the captured TCP probe packet, fills it into the receiving network interface to construct the first TCP reply probe packet, and sends the first TCP reply probe packet through the raw socket Raw Socket. Specifically, in this embodiment, the captured second node receiving network interface is the peer receiving network interface, and the peer receiving network interface is filled into the Recv Iface field.

[0079] The second node obtains the second node sending network interface used by the current path based on the captured TCP probe packet, fills it into the sending network interface to construct the second TCP reply probe packet, and sends the second TCP reply probe packet through the raw socket Raw Socket. Specifically, in this embodiment, the captured second node receiving network interface is the peer sending network interface, and the peer sending network interface is filled into the Send Iface field.

[0080] Preferably, sending the RDMA probe packet through the unreliable datagram mode UD further includes:

[0081] The first node and the second node start an RDMA UD queue pair QP on each network interface, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information including the local identifier LID, queue pair number QPN, and global identifier GID of each RDMA UD queue pair QP on the second node. Specifically, in this embodiment, obtaining the routing information is for subsequent sending of the RDMA probe packet.

[0082] The first node randomly generates several RDMA flow labels for each network interface, and fills the RDMA flow labels into the source port of the RDMA probe packet body to generate the RDMA probe packet body. Specifically, in this embodiment, the flow label is filled into the Src Port field of the RDMA probe packet body, and 10 RDMA probe packets are sent to detect all network interfaces used by each path.

[0083] The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends the RDMA probe packet including the second node routing information and the RDMA probe packet body to the second node through the RDMA work request WR. Specifically, in this embodiment, the RDMA probe packet is sent to the second node through ibv_post_send.

[0084] Preferably, the second node receives an RDMA probe packet and generates an RDMA reply probe packet, further including:

[0085] Only one UD queue pair QP in the RDMA UD queue pair of the second node can receive the RDMA probe packet. Record the receiving network interface of the second node according to the RDMA probe packet. Specifically, in this embodiment, the receiving network interface of the second node is the peer receiving network interface.

[0086] The second node fills the receiving network interface of the second node into the receiving network interface of the RDMA reply probe packet body, and sends the RDMA reply probe packet to the first node by sending an RDMA work request WR. Specifically, in this embodiment, the second node fills the receiving network interface of the second node into the Recv Iface field of the RDMA reply probe packet body.

[0087] S2: The first node obtains path information including source port and network interface information according to the received reply probe packet, and selects a path according to the sending network interface and receiving network interface in the network interface information.

[0088] Preferably, in step S2, the first node obtains path information according to the received reply probe packet, further including:

[0089] If the network connection method is a TCP connection, after the first node receives two reply probe packets, namely the first TCP reply probe packet and the second TCP reply probe packet, the first node can perceive the source port used by each path of the TCP probe packet, the sending network interface of the first node, the receiving network interface of the first node, the sending network interface of the second node, and the receiving network interface of the second node. Specifically, in this embodiment, after the first node receives the first TCP reply probe packet, it can perceive the receiving network interface of the first node, that is, the source receiving network interface. The first node will consider the detection process completed only after receiving two reply probe packets of a certain request probe packet.

[0090] If the network connection method is an RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply probe packet, record the source port for receiving the RDMA reply probe packet. For an RDMA connection, the sending network interface and receiving network interface of a connection are the same. Therefore, the source sending network interface is the same as the source receiving network interface, and the peer sending network interface is the same as the peer receiving network interface. At this time, the first node can perceive the sending network interface of the first node, the receiving network interface of the first node, the sending network interface of the second node, and the receiving network interface of the second node for each path.

[0091] S3: After the first node obtains the path selection result, establish corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establish a multi-path data channel, and perform path health checks regularly.

[0092] Please refer to Figure 4 As shown, after the first node obtains the source sending network interface, source receiving network interface, peer sending network interface, and peer receiving network interface of all the paths corresponding to the probe packets, the TCP / RDMA path probing process is completed. After that, for TCP, the first node can select the source ports of multiple paths to establish TCP connections, where the four network interfaces used between different paths are different;

[0093] For RDMA, the first node can select the flow labels of multiple paths to establish an RDMA reliable connection (Reliable Connection, RC), where the source sending network interface and peer receiving network interface used between different paths are different. After these connections are established, a data channel that can utilize the bandwidth between multiple network interfaces and provide fault tolerance between network cards is established. Please refer to the following example:

[0094] Please refer to Figure 5 As shown, assume that both node A and node B have two network interfaces, and the figure shows the data flow directions of the 2 paths included after the data channel is established. The source sending / receiving network interfaces and peer sending / receiving network interfaces of path 1 are eth1, eth2, eth4, and eth3 respectively, and the source sending / receiving network interfaces and peer sending / receiving network interfaces of path 2 are eth2, eth1, eth3, and eth4.

[0095] Preferably, in step S3, path health checks are performed regularly, which further includes:

[0096] Perform path health checks within a set time, mark the locally used network interfaces with the sending network interfaces of all the current valid paths, so as to sense whether there are unused network interfaces currently; if there are, it will trigger the fault reconnection of multiple paths to perform new path probing, and determine whether the newly probed paths use different network interfaces from the existing valid paths. If there are paths that meet the conditions, use the source port / RDMA flow label to establish a new path. If there are no paths that meet the conditions, end.

[0097] Please refer to Figure 6As shown, within the set time of 1 second, obtain the number of local network interfaces and the number of local network interfaces used by existing TCP / RDMA connections, and determine whether the number of local network interfaces is the same as the number of network interfaces already used locally. If they are the same, it is judged to be normal and the inspection ends. If it is not normal, that is, when a certain path of the data channel fails (such as a network interface being briefly disconnected or a connection being interrupted due to an abnormal switch port), trigger the fault reconnection of multiple paths to perform new path detection. If there is a node A, generate 10 probe packets for each unused network interface. After the path detection ends, determine whether the newly detected path uses a different network interface from the existing valid paths. If there is a path that meets the conditions, establish a new path using the source port of TCP or the flow label of RDMA to supplement the new path to fully utilize the bandwidth between multiple network interfaces. If not, end. Please refer to the following example:

[0098] Please refer to Figure 7 As shown, when both node A and node B have two network interfaces, if the eth2 of node A experiences a brief disconnection of the network interface at this time, it will cause the disconnection of path 2. At this time, the data channel only contains path 1. Based on the sending network interface eth1 used by all current paths (only path 1) and the eth1 and eth2 network interfaces of node A, perform path inspection. It can be found that there is a path failure at this time. At this time, only generate 10 probe packets for the unused eth2 network interface and perform path detection. If eth2 has recovered to a healthy state at this time, and there is a path among the detected multiple paths whose source-end sending / receiving network interface and peer sending / receiving network interfaces are eth2, eth1, eth3, and eth4, a new path can be established based on the source port of TCP or the RDMA flow label of this path.

[0099] It should be noted that in the following scenarios, node A cannot utilize all network interfaces, resulting in frequent triggering of reconnection but unable to actually establish a new path:

[0100] The number of network interfaces of node A is more than that of node B;

[0101] Some network interfaces of node B are always in a disconnected state;

[0102] There is a problem with the configuration of the Link Aggregation Control Protocol of the router.

[0103] To solve the above problems, record the number of consecutive reconnections that trigger a fault reconnection but do not actually complete the path. If this number exceeds the set number, then extend the set time to avoid frequent triggering of reconnections. And as long as a disconnection is sent on one path, the set time will be reset to the initial set time, and the number of consecutive reconnections will be set to 0, so as to quickly complete all paths. Specifically, in this embodiment, if the number of consecutive reconnections exceeds 60 times, then the interval of path health check will be adjusted to 60 seconds to avoid frequent triggering of reconnections. At the same time, as long as a disconnection is sent on one path, the interval will be reset to 1 second.

[0104] Please refer to Figure 8 As shown, node A has three network interfaces, eth1, eth2, and eth5, while node B only has two network interfaces, eth3 and eth4. At this time, the data channel from node A to node B will at most only contain 2 paths. The reason is that even if node A detects a new path using eth5 as the source sending network interface, since the peer receiving network interface must be the already used eth3 or eth4, this path using eth5 will not actually be used to establish a new path. By avoiding the logic of frequent reconnection triggering, the number of reconnection triggers can be greatly reduced in the above scenario.

[0105] This embodiment introduces a path selection mechanism based on probe packets to ensure that each TCP / RDMA data path will select different network interfaces when being established, avoiding the situation where network traffic is concentrated on a few network interfaces. By using multiple probes with different source ports, the node can obtain the source sending network interface, source receiving network interface, peer sending network interface, and peer receiving network interface of each path, thus ensuring that different network interfaces are used among multiple paths, avoiding the bandwidth bottleneck of a single network interface, and ensuring more efficient bandwidth utilization.

[0106] This embodiment provides the fault redundancy ability between network cards. In the existing RDMA data channel design, it is impossible to provide the fault redundancy ability between network cards while utilizing the bandwidth of multiple network interfaces. Through the multi-path mechanism of the data channel in this embodiment, when a network interface of a certain network card fails, the path using the network interface of other network cards can still work normally, thus avoiding communication interruption and improving the availability and fault tolerance of the system.

[0107] This embodiment supports upper-layer autonomous load balancing. In the existing balance-slb and balance-tcp Bond modes, load balancing usually relies on static hashing algorithms. Since the hashing results cannot be adjusted, load imbalance occurs between network interfaces, and it cannot be autonomously adjusted by the upper layer. This embodiment can provide the upper layer with the network interface information used by each path, and the upper layer can autonomously perform information statistics and load balancing. Bond modes such as balance-slb or balance-tcp perform load balancing based on OvSBond and cannot be optimized according to the dynamic changes in actual traffic. This embodiment can support a more flexible mechanism for upper-layer load balancing. Specifically, the upper-layer application can monitor the data traffic on different paths in the data channel, sense key performance indicators such as the bandwidth utilization rate, latency, and load of each path. Based on these real-time feedbacks, the upper-layer custom load balancing mechanism can actively allocate new traffic to the path with a lower current bandwidth utilization rate, thereby achieving load balancing perceived by the upper layer.

[0108] Embodiment 2

[0109] The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the method for multi-path selection of a data channel for distributed storage in an embodiment of the present invention.

[0110] It can be understood that for the method for multi-path selection of a data channel for distributed storage for the aforementioned bus address, if it is implemented in the form of software function modules and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0111] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0112] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. Any technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multi-path selection method for data channels oriented to distributed storage, characterized in that The steps include the following: S1: The first node sends a probe packet with a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network interface, and a receiving network interface according to the network connection mode. The second node receives the probe packet and generates a reply probe packet; S2: The first node obtains path information including the source port and network interface information based on the received reply probe packet, and selects a path according to the sending network interface and the receiving network interface in the network interface information; S3: After the first node obtains the path selection result, it establishes corresponding network TCP / RDMA connections based on the source port / RDMA flow label of the selected multiple paths, that is, establishes a multi-path data channel, and periodically performs path health checks; In step S2, the first node obtains path information based on the received reply probe packet, which further includes: If the network connection mode is a TCP connection, after the first node receives two reply probe packets, namely the first TCP reply probe packet and the second TCP reply probe packet, the first node can sense the source port used by each path of the TCP probe packet, the sending network interface of the first node, the receiving network interface of the first node, the sending network interface of the second node, and the receiving network interface of the second node; If the network connection mode is an RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply probe packet, it records the source port of the received RDMA reply probe packet. For an RDMA connection, the sending network interface and the receiving network interface of a connection are the same. Therefore, the source sending network interface is the same as the source receiving network interface, and the peer sending network interface is the same as the peer receiving network interface. At this time, the first node can sense the sending network interface of the first node, the receiving network interface of the first node, the sending network interface of the second node, and the receiving network interface of the second node for each path.

2. The multi-path selection method for data channels oriented to distributed storage according to claim 1, wherein In step S1, the first node sends a probe packet according to the network connection mode, which further includes: If the network connection mode is a TCP connection, the TCP probe packet is sent through a raw socket. The TCP probe packet includes a TCP header and a TCP probe packet body; If the network connection mode is an RDMA connection, the RDMA probe packet is sent through an unreliable datagram mode UD. The RDMA probe packet includes routing information and the RDMA probe packet body.

3. The multi-path selection method for data channels for distributed storage according to claim 2, wherein Sending the TCP probe packet through a raw socket further includes: The first node specifies the second node IP and the second node port through the sendto function of the raw socket, and fills the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header, and the Ethernet header; The first node generates a plurality of the TCP probe packet bodies for each network interface according to different source ports, and sends the TCP probe packet according to the TCP header and the TCP probe packet body.

4. The multi-path selection method for data channels oriented to distributed storage according to claim 3, wherein The second node receives the TCP probe packet and generates a TCP reply probe packet, which further includes: The second node starts the packet capture tool to capture the TCP probe packet, and uses a filter to identify the version number. If the identification result meets the requirements, it receives the TCP probe packet; if the identification result does not meet the requirements, it discards the TCP probe packet. The second node obtains the second node receiving network interface used by the current path based on the captured TCP probe packet, fills the second node receiving network interface into the receiving network interface to construct the first TCP reply probe packet, and sends the first TCP reply probe packet through the raw socket RawSocket. The second node obtains the second node sending network interface used by the current path based on the captured TCP probe packet, fills the second node sending network interface into the sending network interface to construct the second TCP reply probe packet, and sends the second TCP reply probe packet through the raw socket RawSocket.

5. The multi-path selection method for data channels for distributed storage according to claim 2, characterized in that, Sending the RDMA probe packet through the unreliable datagram mode UD further includes: The first node and the second node start an RDMA UD queue pair QP on each network interface, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information of each RDMA UD queue pair QP on the second node, including the local identifier LID, the queue pair number QPN, and the global identifier GID. The first node randomly generates several of the RDMA flow labels for each network interface, and fills the RDMA flow labels into the source port of the RDMA probe packet body to generate the RDMA probe packet body. The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends the RDMA probe packet including the second node routing information and the RDMA probe packet body to the second node through the RDMA work request WR.

6. The data channel multipath selection method for distributed storage according to claim 4, wherein The second node receives the RDMA probe packet and generates an RDMA reply probe packet, further including: Only one UD queue pair QP in the RDMA UD queue pair QP of the second node can receive the RDMA probe packet, and the second node receiving network interface is recorded according to the RDMA probe packet. The second node fills the second node receiving network interface into the receiving network interface of the RDMA reply probe packet body, and sends the RDMA reply probe packet to the first node by sending an RDMA work request WR.

7. The multi-path selection method for data channels oriented to distributed storage according to claim 6, characterized in that In step S3, performing path health checks regularly further includes: Performing path health checks within a set time, using the sending network interfaces of all current valid paths to mark the locally used network interfaces, so as to sense whether there are unused network interfaces currently; if there are, it will trigger the failure reconnection of multiple paths to perform new path detection, and determine whether the newly detected path uses a different network interface from the existing valid paths. If there are paths that meet the conditions, use the source port / the RDMA flow label to establish a new path. If there are no paths that meet the conditions, end.

8. The data channel multipath selection method for distributed storage according to claim 7, wherein Record the consecutive reconnection times when a fault reconnection is triggered but the path is not actually replenished. If this number exceeds the set number, then extend the set time to avoid frequent triggering of reconnections. And as long as a disconnection occurs on one path, the set time will be reset to the initial set time, and the consecutive reconnection times will be set to 0, so as to quickly complete all paths.

9. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the data channel multipath selection method for distributed storage according to any one of claims 1 to 8.

Images