Distributed storage-oriented data channel multi-path selection method

Abstract

The invention relates to the technical field of network communication, in particular to a distributed storage-oriented data channel multi-path selection method. Comprising the following steps: S1, a first node sends a probe packet having a probe packet header and a probe packet body including a version number, a probe packet type, a probe packet identifier, a source port, a sending network port and a receiving network port according to a network connection mode, and a second node receives the probe packet and generates a reply probe packet; s2, the first node obtains path information including a source port, a sending network port and a receiving network port according to the reply probe packet, and selects a path; and S3, the first node obtains a path selection result, establishes a corresponding network TCP / RDMA connection according to the source port / RDMA flow labels of the plurality of selected paths, that is, establishes a data channel of the plurality of paths, and regularly performs path health examination. According to the invention, TCP / RDMA path detection is realized, a plurality of random source ports are used for detection, the network ports used by each path are ensured to be different, and the bandwidth among the plurality of network ports can be completely utilized.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a data channel multi-path selection method and storage medium oriented to distributed storage. Background Art

[0002] With the rapid development of information technology, the amount of data is showing an explosive growth trend. Traditional single-machine storage methods can no longer meet the needs of storing, managing and efficiently accessing massive amounts of data, and distributed storage systems have emerged. It stores data in multiple physical nodes connected by a network, and through collaborative work, provides unified data storage services to the outside world, with many advantages such as scalability, high reliability, and high performance. In a distributed storage system, multiple physical nodes in a cluster rely on the network to communicate, and data channels are responsible for data transmission between nodes through the network.

[0003] Currently, the data channel between two nodes usually relies on a single network connection, that is, a single path. The common connection methods of the path are TCP and RDMA. However, the performance of the data channel of a single path is limited by the bandwidth of a single network port, and the failure of a single network port will cause data transmission interruption, which in turn affects the availability and overall performance of the cluster. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and provide a data channel multi-path selection method for distributed storage, comprising the following steps: S1: The first node sends a detection packet having a detection packet header and a detection packet body including a version number, a detection packet type, a detection packet identifier, a source port, a sending network port, and a receiving network port according to a network connection mode, and the second node receives the detection packet and generates a reply detection packet; S2: The first node obtains path information including source port and network port information according to the received reply detection packet, and selects a path according to the sending network port and the receiving network port in the network port information; S3: After the first node obtains the path selection result, it establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes multi-path data channels, and performs path health checks regularly.

[0005] Preferably, in step S1, the first node sends a detection packet according to the network connection mode, further comprising: If the network connection mode is a TCP connection, a TCP detection packet is sent through a raw socket, wherein the TCP detection packet includes a TCP header and a TCP detection packet body; If the network connection mode is RDMA connection, an RDMA detection packet is sent through an unreliable datagram mode UD, and the RDMA detection packet includes routing information and the RDMA detection packet body.

[0006] Preferably, sending a TCP detection packet through a raw socket further includes: The first node specifies the second node IP and the second node port through the sendto function of the raw socket, and fills the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header and the Ethernet header; The first node generates a plurality of the TCP detection packet bodies for each network port according to different source ports, and sends the TCP detection packet according to the TCP header and the TCP detection packet body.

[0007] Preferably, the second node receives the TCP detection packet and generates a TCP reply detection packet, further comprising: The second node starts a packet capture tool to capture the TCP detection packet, and identifies the version number through a filter, and receives the TCP detection packet if the identification result meets the requirements, and discards the TCP detection packet if the identification result does not meet the requirements; The second node obtains the second node receiving network port used by the current path according to the captured TCP detection packet, fills the receiving network port according to the second node receiving network port to construct a first TCP reply detection packet, and sends the first TCP reply detection packet through the raw socket RawSocket; The second node obtains the second node sending network port used by the current path according to the captured TCP detection packet, fills the sending network port into the sending network port according to the second node sending network port to construct a second TCP reply detection packet, and sends the second TCP reply detection packet through the raw socket RawSocket.

[0008] Preferably, sending the RDMA detection packet through the unreliable datagram mode UD further includes: The first node and the second node start an RDMA UD queue pair QP on each network port, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information including a local identifier LID, a queue pair number QPN and a global identifier GID of each of the RDMA UD queue pairs QP on the second node; The first node randomly generates a plurality of the RDMA flow labels for each network port, and fills the RDMA flow labels into the source port of the RDMA detection packet to generate an RDMA detection packet; The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends the RDMA detection packet including the second node routing information and the RDMA detection packet body to the second node through the RDMA work request WR.

[0009] Preferably, the second node receives the RDMA detection packet and generates an RDMA reply detection packet, further comprising: Only one UD queue pair QP among the RDMA UD queue pairs QP of the second node can receive the RDMA detection packet, and the receiving network port of the second node is recorded according to the RDMA detection packet; The second node fills the receiving network port of the second node to the receiving network port of the RDMA reply detection packet body, and sends the RDMA reply detection packet to the first node by sending an RDMA work request WR.

[0010] Preferably, in step S2, the first node obtains the path information according to the received reply detection packet, further comprising: If the network connection mode is a TCP connection, after the first node receives the two reply detection packets of the first TCP reply detection packet and the second TCP reply detection packet, the first node can perceive the source port used by each path of the TCP detection packet, the first node sending network port, the first node receiving network port, the second node sending network port, and the second node receiving network port; If the network connection mode is RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply detection packet, it records the source port for receiving the RDMA reply detection packet. For RDMA connection, the sending network port and receiving network port of a connection are the same, so the source sending network port is the same as the source receiving network port, and the opposite sending network port is the same as the opposite receiving network port. At this time, the first node can perceive the first node sending network port, the first node receiving network port, the second node sending network port and the second node receiving network port of each path.

[0011] Preferably, in step S3, regularly performing a path health check further includes: Perform a path health check within the set time, and use the sending network ports of all current valid paths to mark the local used network ports, so as to sense whether there are any unused network ports at present; if so, it will trigger multi-path fault reconnection to perform new path detection, and determine whether the new detection path uses a different network port from the existing valid path. If there is a path that meets the conditions, use the source port / the RDMA stream label to establish a new path. If there is no path that meets the conditions, end the process.

[0012] Preferably, the number of consecutive reconnections that trigger fault reconnection but do not actually fill the path is recorded. If the number exceeds the set number, the set time is extended to avoid frequent triggering of reconnection. As long as one path sends a disconnection, the set time will be reset to the initial set time and the number of consecutive reconnections will be set to 0, so as to quickly fill all paths.

[0013] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data channel multi-path selection method for distributed storage as described in any one of the embodiments.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes TCP / RDMA path detection by sending a detection packet having a detection packet header and a detection packet body including a version number, a detection packet type, a detection packet identifier, a source port, a sending network port and a receiving network port through a first node according to a network connection mode, and a second node receives the detection packet and generates a reply detection packet.

[0015] The first node of the present invention obtains network port information including the source port, the sending network port and the receiving network port according to the received reply detection packet, and selects a path according to the network port information, so as to realize detection by using multiple random source ports. The first node can obtain all network ports used by multiple paths and ensure that the network ports used by each path are different.

[0016] The present invention establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes multi-path data channels, and regularly performs path health checks to utilize all current valid paths, thereby sensing whether there are any network ports that have not been used, and adding new paths to fully utilize the bandwidth between multiple network ports. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0018] Figure 1 A flow chart of a data channel multi-path selection method for distributed storage according to the present invention; Figure 2 It is a TCP detection flow chart of the data channel multi-path selection method for distributed storage of the present invention; Figure 3 It is a flow chart of RDMA detection of a data channel multi-path selection method for distributed storage according to the present invention; Figure 4 A data channel establishment flow chart for a data channel multi-path selection method for distributed storage according to the present invention; Figure 5 An example diagram is established for the data channel of the distributed storage-oriented data channel multi-path selection method of the present invention; Figure 6 A path checking flow chart of a data channel multi-path selection method for distributed storage according to the present invention; Figure 7 This is an example diagram of path checking of the data channel multi-path selection method for distributed storage of the present invention; Figure 8 This is an example diagram of a frequently triggered reconnection scenario of the data channel multi-path selection method for distributed storage of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0020] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0021] Example 1 In order to solve the problem of poor performance and insufficient fault tolerance of a single network port under TCP connection, the current solution usually uses link aggregation (Bonding) technology to improve the bandwidth and reliability of network connection. Link aggregation technology can combine multiple network ports into a logical link. This technology can ensure that even if a network port fails, other network ports can still bear the traffic, thereby avoiding network interruption and enhancing system availability. Among them, common modes of link aggregation include active-backup, balance-slb and balance-tcp. Although link aggregation can theoretically provide the bandwidth of multiple network ports, in actual use, the active-backup mode only supports active-backup switching and cannot use the bandwidth of multiple network ports at the same time; although the balance-slb and balance-tcp modes support load balancing, the path selection depends on the hash result. Most of the data flow may be allocated to some network ports, and the uniform distribution of data flow cannot be ensured. Especially when the number of connections is small, it is easy for some network ports to be overloaded while other network ports are idle. This load imbalance problem limits the bandwidth utilization of multiple network ports and affects the overall network performance.

[0022] Taking Ceph as an example, when TCP connections are used as data channels in a multi-port environment, link aggregation technology is usually relied on to improve bandwidth and availability. For example, when using the balance-tcp mode, the distribution of traffic depends on the hash algorithm (such as source IP, destination IP, port, etc.). In order to fully utilize the bandwidth of all ports, multiple connections are usually required. However, due to the randomness of hash-based load balancing, it is impossible to ensure that the bandwidth of each port is fully utilized, or the load balancing effect between ports is difficult to ensure. This unbalanced load distribution will affect the overall performance of the system, especially in high-concurrency read and write scenarios, where bandwidth cannot be maximized.

[0023] Compared with TCP connection, RDMA connection can provide lower latency and higher throughput, but in a multi-port environment, existing data channels using RDMA connection still have problems with insufficient bandwidth utilization and insufficient fault tolerance. Many existing data channels using RDMA connection only support communication through a single port. For example, Nutanix only supports establishing RDMA connection through a single port. If the port fails, the data channel will degenerate to using TCP connection for communication, resulting in a significant decrease in data transmission performance and throughput, which cannot meet the requirements of high reliability and high performance. In addition, this solution cannot achieve bandwidth aggregation in a multi-port environment, and cannot give full play to the performance advantages of combining a multi-port environment with RDMA.

[0024] In vSAN, although the data channel connected by RDMA supports NIC failover, it also lacks support for more complex NIC teaming such as Link Aggregation Control Protocol (LACP) or IP address hashing. When a NIC fails, the system will automatically switch to the backup NIC to continue communication, thereby maintaining the stability of data transmission. However, this solution also cannot achieve effective utilization of multi-port bandwidth.

[0025] Therefore, although RDMA provides the advantages of low latency and high bandwidth, in a multi-network port environment, the existing data channel design using RDMA connection fails to effectively solve the bandwidth aggregation and fault tolerance problems, resulting in the failure to fully utilize the performance of RDMA. This limitation of insufficient bandwidth and fault tolerance makes the existing RDMA data channel solution unable to meet the high availability and high performance requirements of distributed storage systems, and fails to fully utilize the low latency and high bandwidth advantages of RDMA in a multi-network port environment, limiting the application potential of the system in high availability and high performance scenarios.

[0026] In order to solve the above problems, this embodiment has the following prerequisites: 1. Bond Type Currently, Linux Bond and Open vSwitch (OvS) Bond both support network link aggregation on Linux systems, which can merge multiple network ports into one logical link, thereby increasing bandwidth and providing redundancy. However, Linux Bond has certain limitations in supporting RDMA devices. Specifically, in Linux Bond RDMA mode, only multiple network ports of the same network card can be aggregated, and port aggregation cannot be performed across different network cards, nor can fault tolerance between network cards be provided. After adopting Linux Bond, the upper-layer application software can only see one aggregated virtual network device, and can no longer specify in more detail which network card the data is sent / received through. Therefore, Linux Bond cannot meet the requirements of this embodiment, which requires that OvS Bond must be used for network link aggregation, and that ports between different network cards must be aggregated.

[0027] 2. Bond Mode Ovs Bond mainly has several modes: active-backup, balance-slb and balance-tcp. Among them, the active-backup mode only uses active slave to send network packets, and cannot use the bandwidth of multiple network ports; the balance-slb mode is based on the source MAC hash. Considering that each node of the distributed storage system usually uses the Bond MAC address for communication after configuring Bond, this mode cannot use multiple network ports; the balance-tcp mode is based on the source and target IP, source and target TCP / UDP ports. The network port for sending network packets is selected based on the hash result. Therefore, as long as the data channel can reasonably establish multiple TCP / RDMA connections, all network ports in Bond can be used for sending.

[0028] 3. Switch configuration When using Ovs Bond and balance-tcp mode, multiple connections established in the data channel can use multiple network ports of the sender, but the receiving network port used by the receiver is controlled by the switch's port aggregation (Port Channel). The switch's port aggregation can generally configure the fields used for hashing. If these fields of multiple connections are different, there is a chance that they will be hashed to different network ports of the receiver. This embodiment requires that the switch's port aggregation be configured to hash based on source-destination-port.

[0029] See also Figure 1 As shown, the data channel multi-path selection method for distributed storage provided in this embodiment ensures that the data channel from the first node (client) to the second node (server) in the distributed storage system can transmit data through multiple TCP or RDMA connections (i.e., multi-paths) in a multi-network port environment. These connection combinations will fully utilize all network ports and link capabilities of the client and the server. The data channel can provide an RDMA cross-network card aggregation mode to provide fault tolerance for network card failures and support flexible control of data traffic load balancing by upper-layer applications, including the following steps: S1: The first node sends a detection packet having a detection packet header and a detection packet body including a version number, a detection packet type, a detection packet identifier, a source port, a sending network port and a receiving network port according to the network connection method. The second node receives the detection packet and generates a reply detection packet. Specifically, in this embodiment, the format of the detection packet body is Version|Type|Message ID|Src Port|Send Iface|Recv Iface|Reserve. TCP and RDMA both use the same format of detection packet body, and this embodiment does not limit the format and can be changed as needed.

[0030] Preferably, the meanings of the fields in the detection packet body are as follows: Version: version number, used to distinguish from other TCP packets and support subsequent upgrades; Type: The type of the detection packet, indicating whether it is a request or a reply; Message ID: An identifier of a request probe packet. The reply probe packet uses the same ID as the probe packet. Src Port: The source TCP port used by the detection packet; Send Iface: used to reply to the detection packet and transmit the sending network port of the other end; Recv Iface: used to reply to the detection packet and transmit the receiving network port of the other end; Reserve: Reserved field.

[0031] See also Figure 2 and Figure 3 As shown, in step S1, the first node sends a detection packet according to the network connection mode, further comprising: If the network connection mode is TCP connection, the first node needs to start a packet capture tool for capturing the detection packet and the reply detection packet before sending the detection packet, and then send the TCP detection packet through the raw socket. The TCP detection packet includes a TCP header and a TCP detection packet body; If the network connection mode is RDMA connection, an RDMA detection packet is sent through the unreliable datagram mode UD. The RDMA detection packet includes routing information and an RDMA detection packet body. Specifically, in this embodiment, the RDMA detection packet cannot use the original socket to send the detection packet and use the packet capture tool to receive it. The reason is that the packet capture tool needs the kernel to participate in capturing the RDMA detection packet, which will cause a large performance overhead, and the current packet capture tool does not support non-blocking mode to capture RDMA detection packets.

[0032] Preferably, sending a TCP detection packet through a raw socket further includes: The first node specifies the second node IP and the second node port through the sendto function of the raw socket, and fills the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header and the Ethernet header; The first node generates several TCP detection packets for each network port according to different source ports, and sends TCP detection packets according to the TCP header and the TCP detection packet body. Specifically, in this embodiment, 10 TCP detection packets are sent, and the source ports of the TCP detection packets are different. All network ports used by each path are detected, and the detection packets are captured by a packet capture tool to obtain the first node sending network port used by the current path, that is, the source end sending network port; Preferably, the second node receives the TCP detection packet and generates a TCP reply detection packet, further comprising: The second node starts the packet capture tool to capture the TCP detection packet and identifies the version number through the filter. If the identification result meets the requirements, the TCP detection packet is received. If the identification result does not meet the requirements, the TCP detection packet is discarded. The second node obtains the second node receiving network port used by the current path according to the captured TCP detection packet, fills the receiving network port according to the second node receiving network port to construct a first TCP reply detection packet, and sends the first TCP reply detection packet through the raw socket Raw Socket. Specifically, in this embodiment, the captured second node receiving network port is the opposite end receiving network port, and the opposite end receiving network port is filled into the Recv Iface field; The second node obtains the second node sending network port used by the current path based on the captured TCP detection packet, fills the sending network port into the sending network port according to the second node sending network port to construct a second TCP reply detection packet, and sends the second TCP reply detection packet through the raw socket Raw Socket. Specifically, in this embodiment, the captured second node receiving network port is the sending network port of the opposite end, and the sending network port of the opposite end is filled into the Send Iface field.

[0033] Preferably, sending the RDMA detection packet through the unreliable datagram mode UD further includes: The first node and the second node start an RDMA UD queue pair QP on each network port, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information including the local identifier LID, the queue pair number QPN and the global identifier GID of each RDMA UD queue pair QP on the second node. Specifically, in this embodiment, the routing information is obtained so as to subsequently send an RDMA detection packet; The first node randomly generates several RDMA flow labels for each network port, and fills the RDMA flow labels into the source port of the RDMA detection packet body to generate an RDMA detection packet body. Specifically, in this embodiment, the flow label is filled into the Src Port field of the RDMA detection packet body, and 10 RDMA detection packets are sent to detect all network ports used by each path; The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends an RDMA detection packet including the second node routing information and the RDMA detection packet body to the second node through the RDMA work request WR. Specifically, in this embodiment, the RDMA detection packet is sent to the second node through ibv_post_send.

[0034] Preferably, the second node receives the RDMA detection packet and generates an RDMA reply detection packet, further comprising: Only one UD queue pair QP among the RDMA UD queue pairs QP of the second node can receive the RDMA detection packet, and the receiving network port of the second node is recorded according to the RDMA detection packet. Specifically, in this embodiment, the receiving network port of the second node is the receiving network port of the opposite end; The second node fills the second node receiving network port into the receiving network port of the RDMA reply detection packet body, and sends the RDMA reply detection packet to the first node by sending an RDMA work request WR. Specifically, in this embodiment, the second node fills the second node receiving network port into the Recv Iface field of the RDMA reply detection packet body.

[0035] S2: The first node obtains path information including source port and network port information according to the received reply detection packet, and selects a path according to the sending network port and the receiving network port in the network port information.

[0036] Preferably, in step S2, the first node obtains the path information according to the received reply detection packet, further comprising: If the network connection mode is a TCP connection, after the first node receives the two reply detection packets of the first TCP reply detection packet and the second TCP reply detection packet, the first node can sense the source port used by each path of the TCP detection packet, the first node sending network port, the first node receiving network port, the second node sending network port, and the second node receiving network port. Specifically, in this embodiment, after the first node receives the first TCP reply detection packet, it can sense the first node receiving network port, that is, the source receiving network port. The first node will consider that the detection process is completed only after receiving two reply detection packets of a request detection packet; If the network connection mode is RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply detection packet, it records the source port of receiving the RDMA reply detection packet. For RDMA connection, the sending network port and receiving network port of a connection are the same, so the source sending network port is the same as the source receiving network port, and the peer sending network port is the same as the peer receiving network port. At this time, the first node can perceive the first node sending network port, the first node receiving network port, the second node sending network port and the second node receiving network port of each path.

[0037] S3: After the first node obtains the path selection result, it establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes multi-path data channels, and performs path health checks regularly.

[0038] See also Figure 4 As shown, after the first node obtains the source end sending network port, the source end receiving network port, the opposite end sending network port, and the opposite end receiving network port of the paths corresponding to all detection packets, the TCP / RDMA path detection process is completed. After that, for TCP, the first node can select the source ports of multiple paths to establish a TCP connection, where the four network ports used between different paths are different; For RDMA, the first node can select multiple path flow labels to establish an RDMA reliable connection (RC), where the source sending network port and the peer receiving network port used by different paths are different. After these connections are established, the bandwidth between multiple network ports can be utilized, and the data channel that can provide fault tolerance between network cards is established. Please refer to the following example: See also Figure 5 As shown in the figure, it is assumed that both node A and node B have two network ports, and the figure shows the data flow direction of the two paths after the data channel is established. The source send / receive network ports and the peer send / receive network ports of path 1 are eth1, eth2, eth4 and eth3 respectively, and the source send / receive network ports and the peer send / receive network ports of path 2 are eth2, eth1, eth3 and eth4.

[0039] Preferably, in step S3, regularly performing a path health check further includes: Perform path health check within the set time, use the sending network ports of all current valid paths to mark the local used network ports, so as to sense whether there are unused network ports at present; if so, it will trigger multi-path fault reconnection to detect new paths, and determine whether the new detection path uses a different network port from the existing valid path. If there is a path that meets the conditions, use the source port / RDMA flow label to establish a new path. If there is no path that meets the conditions, end it.

[0040] See also Figure 6As shown, within the set time of 1 second, the number of local network ports and the number of local network ports used by existing TCP / RDMA connections are obtained, and it is determined whether the number of local network ports is the same as the local used network ports. If they are the same, it is determined to be normal and the inspection ends. If it is not normal, that is, when a path of the data channel fails (such as a temporary loss of connection of the network port or an abnormal switch port causes the connection to be interrupted, the multi-path fault reconnection is triggered to perform new path detection. If there is a node A, 10 detection packets are generated for each unused network port. After the path detection is completed, it is determined whether the newly detected path uses a different network port from the existing valid path. If there is a path that meets the conditions, a new path is established using the TCP source port or the RDMA flow label, and the new path is supplemented to fully utilize the bandwidth between multiple network ports. If it does not exist, it ends. Please refer to the following example: See also Figure 7 As shown in the figure, when both node A and node B have two network ports, if the network port eth2 of node A is temporarily disconnected at this time, path 2 will be disconnected. At this time, the data channel only contains path 1. According to the sending network port eth1 used by all current paths (only path 1) and the eth1 and eth2 network ports of node A, a path check can be performed. It can be found that a path is faulty at this time. At this time, 10 detection packets can be generated only for the unused eth2 network port and path detection can be performed. If eth2 has recovered at this time, and there is a path among the multiple detected paths whose source sending / receiving network port and the peer sending / receiving network port are eth2, eth1, eth3 and eth4, a new path can be established based on the TCP source port or RDMA flow label of the path.

[0041] It should be noted that the following scenarios will cause Node A to be unable to use all network ports, thus frequently triggering reconnection but failing to actually establish a new path: Node A has more network ports than node B; Some network ports of Node B are always disconnected; There is a problem with the Link Aggregation Control Protocol configuration of the router.

[0042] In order to solve the above problem, the number of consecutive reconnections that trigger fault reconnection but do not actually fill the path is recorded. If the number exceeds the set number, the set time will be extended to avoid frequent triggering of reconnection. As long as one path sends a disconnection, the set time will be reset to the initial set time, and the number of consecutive reconnections will be set to 0, so as to quickly fill all paths. Specifically, in this embodiment, if the number of consecutive reconnections exceeds 60 times, the interval of the path health check will be adjusted to 60 seconds to avoid frequent triggering of reconnection. At the same time, as long as one path sends a disconnection, the interval will be reset to 1 second.

[0043] See also Figure 8 As shown in the figure, node A has three network ports, eth1, eth2, and eth5, while node B has only two network ports, eth3 and eth4. At this time, the data channel from node A to node B will only contain two paths at most. The reason is that even if node A detects a new path using eth5 as the source sending network port, since the receiving network port on the other end must be eth3 or eth4 that has been used, this path using eth5 will not actually be used to establish a new path. By avoiding the logic of frequently triggering reconnection, the number of reconnection triggers can be greatly reduced in the above scenario.

[0044] This embodiment introduces a path selection mechanism based on detection packets to ensure that each TCP / RDMA data path selects a different network port when it is established, thereby avoiding the situation where network traffic is concentrated on a few network ports. By using multiple source ports for detection, the node can obtain the source sending network port, source receiving network port, opposite sending network port and opposite receiving network port of each path, thereby ensuring that the network ports used between multiple paths are different, avoiding the bandwidth bottleneck of a single network port, and ensuring more efficient bandwidth utilization.

[0045] This embodiment provides fault redundancy between network cards. The existing RDMA data channel design cannot provide fault redundancy between network cards while utilizing the bandwidth of multiple network ports. This embodiment uses a multi-path mechanism of the data channel to enable the paths using the network ports of other network cards to work normally when a network port of a certain network card fails, thereby avoiding communication interruption and improving the availability and fault tolerance of the system.

[0046] This embodiment supports autonomous load balancing of the upper layer. The existing balance-slb and balance-tcp Bond modes usually rely on static hash algorithms for load balancing. Since the hash results cannot be adjusted, the load between network ports is unbalanced, and the upper layer cannot make autonomous adjustments. This embodiment can provide the upper layer with the network port information used by each path, and the upper layer can autonomously perform information statistics and load balancing. Bond modes such as balance-slb or balance-tcp perform load balancing based on OvSBond, and cannot be optimized according to the dynamic changes of actual traffic. This embodiment can support a more flexible mechanism for load balancing by the upper layer. Specifically, the upper layer application can perceive the key performance indicators such as bandwidth utilization, delay, and load of each path by monitoring the data traffic on different paths in the data channel. Based on these real-time feedbacks, the upper layer's custom load balancing mechanism can actively allocate new traffic to paths with lower current bandwidth utilization, thereby achieving load balancing perceived by the upper layer.

[0047] Example 2 The present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, enables the one or more processors to execute the steps of a data channel multi-path selection method for distributed storage in an embodiment of the present invention.

[0048] It is understandable that, for the aforementioned bus address-oriented distributed storage data channel multi-path selection method, if it is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0049] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0050] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A data channel multi-path selection method for distributed storage, characterized in that: The following steps are involved: S1: The first node sends a detection packet having a detection packet header and a detection packet body including a version number, a detection packet type, a detection packet identifier, a source port, a sending network port, and a receiving network port according to a network connection mode, and the second node receives the detection packet and generates a reply detection packet; S2: The first node obtains path information including source port and network port information according to the received reply detection packet, and selects a path according to the sending network port and the receiving network port in the network port information; S3: After the first node obtains the path selection result, it establishes corresponding network TCP / RDMA connections according to the source ports / RDMA flow labels of the selected multiple paths, that is, establishes multi-path data channels, and performs path health checks regularly.

2. The data channel multi-path selection method for distributed storage according to claim 1, characterized in that: In step S1, the first node sends a detection packet according to the network connection mode, further comprising: If the network connection mode is a TCP connection, a TCP detection packet is sent through a raw socket, wherein the TCP detection packet includes a TCP header and a TCP detection packet body; If the network connection mode is RDMA connection, an RDMA detection packet is sent through an unreliable datagram mode UD, and the RDMA detection packet includes routing information and the RDMA detection packet body.

3. The data channel multi-path selection method for distributed storage according to claim 2, characterized in that: Sending a TCP detection packet through a raw socket further includes: The first node specifies the second node IP and the second node port through the sendto function of the raw socket, and fills the IP header and the Ethernet header by itself through the network protocol stack, and obtains the TCP header according to the second node IP, the second node port, the IP header and the Ethernet header; The first node generates a plurality of the TCP detection packet bodies for each network port according to different source ports, and sends the TCP detection packet according to the TCP header and the TCP detection packet body.

4. The data channel multi-path selection method for distributed storage according to claim 3, characterized in that: The second node receives the TCP detection packet and generates a TCP reply detection packet, further comprising: The second node starts a packet capture tool to capture the TCP detection packet, and identifies the version number through a filter, and receives the TCP detection packet if the identification result meets the requirements, and discards the TCP detection packet if the identification result does not meet the requirements; The second node obtains the second node receiving network port used by the current path according to the captured TCP detection packet, fills the receiving network port according to the second node receiving network port to construct a first TCP reply detection packet, and sends the first TCP reply detection packet through the raw socket RawSocket; The second node obtains the second node sending network port used by the current path according to the captured TCP detection packet, fills the sending network port into the sending network port according to the second node sending network port to construct a second TCP reply detection packet, and sends the second TCP reply detection packet through the raw socket RawSocket.

5. The data channel multi-path selection method for distributed storage according to claim 2, characterized in that: Sending the RDMA detection packet through the unreliable datagram mode UD further includes: The first node and the second node start an RDMA UD queue pair QP on each network port, and the second node starts a remote procedure call client RPC Server to allow the first node to obtain the second node routing information including a local identifier LID, a queue pair number QPN and a global identifier GID of each of the RDMA UD queue pairs QP on the second node; The first node randomly generates a plurality of the RDMA flow labels for each network port, and fills the RDMA flow labels into the source port of the RDMA detection packet to generate an RDMA detection packet; The first node generates an RDMA work request WR based on the second node routing information and the RDMA flow label, and sends the RDMA detection packet including the second node routing information and the RDMA detection packet body to the second node through the RDMA work request WR.

6. The data channel multi-path selection method for distributed storage according to claim 4, characterized in that: The second node receives the RDMA detection packet and generates an RDMA reply detection packet, further comprising: Only one UD queue pair QP among the RDMA UD queue pairs QP of the second node can receive the RDMA detection packet, and the receiving network port of the second node is recorded according to the RDMA detection packet; The second node fills the receiving network port of the second node to the receiving network port of the RDMA reply detection packet body, and sends the RDMA reply detection packet to the first node by sending an RDMA work request WR.

7. The data channel multi-path selection method for distributed storage according to claims 4 and 6, characterized in that: In step S2, the first node obtains path information according to the received reply detection packet, further comprising: If the network connection mode is a TCP connection, after the first node receives the two reply detection packets of the first TCP reply detection packet and the second TCP reply detection packet, the first node can perceive the source port used by each path of the TCP detection packet, the first node sending network port, the first node receiving network port, the second node sending network port, and the second node receiving network port; If the network connection mode is RDMA connection, after the UD queue pair QP of the first node receives the RDMA reply detection packet, it records the source port for receiving the RDMA reply detection packet. For RDMA connection, the sending network port and receiving network port of a connection are the same, so the source sending network port is the same as the source receiving network port, and the opposite sending network port is the same as the opposite receiving network port. At this time, the first node can perceive the first node sending network port, the first node receiving network port, the second node sending network port and the second node receiving network port of each path.

8. The data channel multi-path selection method for distributed storage according to claim 7, characterized in that: In step S3, a path health check is performed regularly, further comprising: Perform a path health check within the set time, and use the sending network ports of all current valid paths to mark the local used network ports, so as to sense whether there are any unused network ports at present; if so, it will trigger multi-path fault reconnection to perform new path detection, and determine whether the new detection path uses a different network port from the existing valid path. If there is a path that meets the conditions, use the source port / the RDMA stream label to establish a new path. If there is no path that meets the conditions, end the process.

9. The data channel multi-path selection method for distributed storage according to claim 8, characterized in that: The number of consecutive reconnections that trigger fault reconnection but do not actually fill the path is recorded. If the number exceeds the set number, the set time is extended to avoid frequent triggering of reconnection. As long as one path sends a disconnection, the set time will be reset to the initial set time and the number of consecutive reconnections will be set to 0, so as to quickly fill all paths.

10. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data channel multi-path selection method for distributed storage as described in any one of claims 1 to 8.

Images