A method and apparatus for IB network link layer state synchronization

Abstract

The application provides an IB network link layer state synchronization method and device, the method comprising: realizing link state synchronization of a docking device by controlling behaviors of sending and receiving packets in different states; when a local IB device link layer enters a LinkInitialize state from a LinkDown state, sending a link synchronization flow control packet to a peer IB device; after receiving a link synchronization flow control packet or an initialization flow control packet of the peer IB device, sending an initialization flow control packet to the peer IB device; when the local IB device link layer is in a LinkArm, LinkActive or LinkActDefer state and a link synchronization flow control packet of the peer IB device is received, returning a link layer state machine of the local IB device to a LinkDown state for restarting. The technical scheme of the application realizes link layer state synchronization and rapid recovery of a link state after restarting.

Description

Technical Field

[0001] This application belongs to the field of network transmission, and specifically relates to an IB network link layer state synchronization method and apparatus. Background Technology

[0002] InfiniBand (IB) networks are widely used in high-performance computing, big data analytics, and cloud computing due to their advantages such as low latency, high bandwidth, and scalability. The InfiniBand Architecture (IBA) defines a system area network for IB protocol communication. This network can be divided into multiple interconnected subnets, each consisting of end nodes, switches, routers, and a subnet manager. Figure 1 This is a schematic diagram of a subnet topology.

[0003] The subnet manager performs management tasks within the subnet, including maintaining the subnet topology, maintaining the LIDs of subnet devices, maintaining subnet communication routing information, and managing the status of subnet devices. When a new IB device is added to the network or an IB device restarts, the subnet manager needs to update the subnet topology and perform link establishment operations for the IB device. Summary of the Invention

[0004] The purpose of this application is to provide an IB network link layer state synchronization method and apparatus, which aims to achieve link layer state synchronization and rapid recovery of link state after restart.

[0005] According to a first aspect of this application, an IB network link layer state synchronization method is provided, comprising:

[0006] When the local IB device's link layer enters the LinkInitialize state from the LinkDown state, it sends a link synchronization flow control packet to the peer IB device. After receiving the link synchronization flow control packet or initialization flow control packet from the peer IB device, it then sends an initialization flow control packet to the peer IB device.

[0007] When the local IB device's link layer is in the LinkArm, LinkActive, or LinkActDefer state and receives a link synchronization flow control packet from the peer IB device, the local IB device's link layer state machine is returned to the LinkDown state for restart.

[0008] In an optional implementation, the step of sending an initialization flow control packet to the peer IB device after receiving the link synchronization flow control packet or the initialization flow control packet from the peer IB device further includes:

[0009] When the peer IB device receives the link synchronization flow control packet from the local IB device, it will return the link layer state machine of the peer IB device to the LinkDown state to restart and enter the LinkInitialize state, and at the same time return the link synchronization flow control packet to the local IB device.

[0010] When the local IB device receives the link synchronization flow control packet or initialization flow control packet from the peer IB device, it determines that the peer IB device is also in the LinkInitialize state, sends an initialization flow control packet to the peer IB device, and completes the LinkInitialize handshake when it receives the initialization flow control packet returned by the peer device.

[0011] In an optional implementation, after the handshake is completed in the LinkInitialize state, when the IB device receives the CPortState=Arm command from the subnet manager, the link layer state machine of the IB device is controlled to enter the LinkArm state.

[0012] In the LinkArm state, when the IB device receives the CPortState=Active or message activation command from the subnet manager, it controls the IB device's link layer state machine to enter the LinkActive state.

[0013] In an optional implementation, the step of returning the link layer state machine of the local IB device to the LinkDown state for restarting further includes:

[0014] In the LinkArm state, when a link synchronization flow control packet or a physical layer link not established command is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

[0015] In the LinkActive state, when a link synchronization flow control packet or initialization flow control packet is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

[0016] In the LinkActDefer state, when a link synchronization flow control packet or a physical layer link not established command is received from the peer IB device and timed out, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

[0017] According to a second aspect of this application, an IB network link layer state synchronization device is provided, comprising:

[0018] The handshake unit is used to send a link synchronization flow control packet to the peer IB device when the link layer of the local IB device enters the LinkInitialize state from the LinkDown state. After receiving the link synchronization flow control packet or the initialization flow control packet from the peer IB device, it sends an initialization flow control packet to the peer IB device.

[0019] The restart unit is used to restart the link layer state machine of the local IB device when the link layer of the local IB device is in the LinkArm, LinkActive or LinkActDefer state and receives the link synchronization flow control packet of the peer IB device.

[0020] Compared with related technologies, the technical solution of this application has the following advantages:

[0021] Through a more robust IB link-layer state synchronization mechanism, link-layer state synchronization is systematically achieved. It adapts to standard subnet manager behavior, exhibits excellent software compatibility, and eliminates the need for physical layer resets when performing individual hardware resets of the IB device's link layer. This facilitates rapid link recovery after a reset and improves the efficiency of network topology and routing information updates. When the IB device's link layer executes the subnet manager's CPortState=Down command, a state restart is directly and effectively performed, promoting rapid link state recovery and improving network data transmission efficiency. Furthermore, the synchronization of link-layer states prevents functional errors caused by differences in the link-layer states of connected devices, enhancing network stability and reliability.

[0022] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures and processes shown in the description and the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the IB network subnet topology based on relevant technologies.

[0025] Figure 2 This is a schematic diagram of the IB network layer reference model based on relevant technologies.

[0026] Figure 3 This is a schematic diagram of the IB network link layer state machine based on relevant technologies.

[0027] Figure 4 This is a schematic diagram of the communication status of two-end IB devices based on relevant technologies.

[0028] Figure 5 This is a flowchart of an IB network link layer state synchronization method according to an exemplary embodiment of this application.

[0029] Figure 6 This is a schematic diagram of the IB network link layer state machine according to an exemplary embodiment of this application.

[0030] Figures 7-9 This is a schematic diagram of the communication state of a dual-ended IB device according to several exemplary embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The reference model of a single IB device can be divided into the physical layer, link layer, network layer, transport layer, and upper software layer, such as... Figure 2 As shown. IB devices need to establish a physical layer connection first, followed by a link layer connection. Data packets can only be transmitted after both the physical and link layers have completed the connection establishment. The link layer manages the connection establishment state through a link layer state machine, as shown in the image. Figure 3 As shown.

[0033] The IB link layer state machine has five states: LinkDown, LinkInitialize, LinkArm, LinkActive, and LinkActDefer. For a detailed description of each state, please refer to the IB standard protocol. Figure 3In the code, CPort XX represents the subnet manager's command CPortState = XX. For example, CPort Down means CPortState = Down. The subnet manager uses these commands to control the state transitions of the link layer. PHY UP indicates that the physical layer has successfully established a link, and PHY DOWN indicates that the physical layer has not established a link. Remote initialization indicates that the local link layer state machine has returned to the LinkDown state after receiving the initialization flow control packet from the peer in the LinkActive or LinkActDefer state. Packet activation indicates that the LinkArm state has received a packet with a correct CRC. The link layer sends an initialization flow control packet in the LinkInitialize state, sends a normal flow control packet in the LinkArm and LinkActive states, and does not send flow control packets in the LinkDown and LinkActDefer states.

[0034] When a data transmission anomaly occurs at the link layer, the link layer can re-establish the link by returning to the LinkDown state via hardware reset, or by issuing the command CPortState=Down to the link layer through the subnet manager to return to the initial LinkDown state. When one end of the link layer restarts and enters the LinkInitialize state from LinkDown, it sends an initialization flow control packet; when the other end receives the initialization flow control packet in the LinkActive and LinkActDefer states, it will trigger a remote initialization event, causing the other end's state machine to also return to LinkDown for state restart.

[0035] When resetting the link layer, it is usually necessary to reset the underlying physical layer as well. After the reset is complete, the physical layer link needs to be re-established first, followed by the link layer link establishment. Both of these link establishment processes require the subnet manager's involvement. In large network systems, due to the heavy workload of the subnet manager, restoring a single IB device to a normal link state after a reset can take up to several hours, significantly impacting the transmission efficiency of data packets in the network.

[0036] On the other hand, when the subnet manager executes the CPortState=Down command on the local device, if the remote device is in LinkArm state, the local device's link layer state will restart but the remote device's state will not restart. This will cause abnormalities in the link layer flow control credits at both ends, and consequently lead to link layer function malfunctions. Figure 4Taking a scenario where both ends of the device are in LinkArm state at the link layer as an example, when the CPortState=Down command is executed on the local device, the local device will return to the LinkDown state and then enter the LinkInitialize and LinkArm states. However, the remote device will remain in the LinkArm state, resulting in an abnormal situation where one device restarts while the other does not. Similarly, if the local device is in the LinkInitialize or LinkActive state while the remote device is in the LinkArm state, executing the CPortState=Down command on the local device will also result in a link layer anomaly.

[0037] Based on the above analysis, this application proposes an IB network link layer state synchronization method and apparatus. When an IB device at one end restarts its state due to a reset or subnet manager command, the restart and synchronization of the IB device's state at the other end is achieved by adding the sending and receiving of custom link synchronization flow control packets and handshaking. After the link layer enters LinkInitialize from LinkDown, it first sends and receives custom link synchronization flow control packets, then sends and receives initialization flow control packets, and performs handshaking checks in the LinkInitialize state. The custom link synchronization flow control packets can trigger a remote initialization A event, that is, when the other end receives the custom link synchronization flow control packet in the LinkArm / LinkActive / LinkActDefer state, it returns to the LinkDown state.

[0038] See Figure 5 The flowchart provided in this application illustrates the IB network link layer state synchronization method, which includes:

[0039] Step 501: When the local IB device's link layer enters the LinkInitialize state from the LinkDown state, it sends a link synchronization flow control packet to the peer IB device. After receiving the link synchronization flow control packet or initialization flow control packet from the peer IB device, it sends an initialization flow control packet to the peer IB device.

[0040] According to the IB protocol, the format of the link layer flow control packet (FCP) is as follows:

[0041]

[0042] The OP field protocol definition already uses 4'b0000 and 4'b0001, which are the initialization flow control packet OP 1FCP and the normal flow control packet OP 0 FCP, respectively. The custom link synchronization flow control packet in this scheme uses the FCP with the OP field set to 4'b1010, i.e., OP A FCP.

[0043] For example, when the link layer enters the LinkInitialize state from the LinkDown state, it first sends and receives OPAFCP, and then sends and receives the initial flow control packet OP 1FCP. Specifically, the handshake is performed as follows: Upon entering the LinkInitialize state, OPAFCP is first sent to the peer; after receiving OPAFCP or OP 1FCP from the peer in the LinkInitialize state, OP 1FCP is sent, and the handshake is recorded as complete upon receiving OP 1FCP.

[0044] Step 502: When the local IB device's link layer is in the LinkArm, LinkActive, or LinkActDefer state and receives a link synchronization flow control packet from the peer IB device, the local IB device's link layer state machine is returned to the LinkDown state for restart.

[0045] For example, when the LinkArm / LinkActive / LinkActDefer state receives OPAFCP, it triggers a remote initialization A event, returning the link layer state machine to the LinkDown state for restart.

[0046] The link layer state machine and state transition scheme are as follows: Figure 6 and 7 As shown.

[0047] In the LinkInitialize state, OPAFCP is sent and received first, followed by OP1FCP initialization flow control packets to complete the LinkInitialize handshake. In the subsequent LinkArm and LinkActive states, OP0 FCP normal flow control packets are sent and received.

[0048] When the peer device is in LinkArm state, executing CPortState=Down to restart the local device can also restart the peer device. Figure 8 As shown. After receiving OPAFCP in the LinkArm state, the peer device will trigger the remote initialization A event, return to the LinkDown state, and then enter the LinkInitialize state. When both devices are in the LinkInitialize state, they will perform an FCP handshake. After the handshake is completed, they can enter the LinkArm state, thus restarting the state of both ends.

[0049] The above explanation uses the example of both ends being in LinkArm state. Even when the local end is in LinkInitialize or LinkActive state while the remote end is in LinkArm state, the link layer anomaly issue of executing the CPortState=Down command can still be resolved. Therefore, it can compensate for the vulnerability of the CPortState=Down command protocol, thereby ensuring the effectiveness of the subnet manager in executing the CPortState=Down command.

[0050] When resetting the link layer, it is not necessary to reset the physical layer to restart the link layers at both ends.

[0051] Figure 9 Taking the case where both ends are in the LinkActive state as an example, a link layer reset operation is performed on the local device. After the local device enters the LinkDown state, it enters the LinkInitialize state and sends an OP AFCP. After receiving the OP AFCP in the LinkActive state, the remote device will trigger the remote initialization A event, return to the LinkDown state, and then enter the LinkInitialize state. When both devices are in the LinkInitialize state, an FCP handshake is performed. After the handshake is completed, it can enter the LinkArm state and then enter the LinkActive state, thus restarting the state of both ends.

[0052] The reset and restart principles for other link layer states are similar.

[0053] As can be seen, the IB network link layer state synchronization method proposed in this application has the following advantages compared with related technologies:

[0054] Through a more robust IB link-layer state synchronization mechanism, link-layer state synchronization is systematically achieved. It adapts to standard subnet manager behavior, exhibits excellent software compatibility, and eliminates the need for physical layer resets when performing individual hardware resets of the IB device's link layer. This facilitates rapid link recovery after a reset and improves the efficiency of network topology and routing information updates. When the IB device's link layer executes the subnet manager's CPortState=Down command, a state restart is directly and effectively performed, promoting rapid link state recovery and improving network data transmission efficiency. Furthermore, the synchronization of link-layer states prevents functional errors caused by differences in the link-layer states of connected devices, enhancing network stability and reliability.

[0055] Accordingly, this application provides, in a second aspect, an IB network link layer state synchronization apparatus, comprising:

[0056] The handshake unit is used to send a link synchronization flow control packet to the peer IB device when the link layer of the local IB device enters the LinkInitialize state from the LinkDown state. After receiving the link synchronization flow control packet or the initialization flow control packet from the peer IB device, it sends an initialization flow control packet to the peer IB device.

[0057] The restart unit is used to restart the link layer state machine of the local IB device when the link layer of the local IB device is in the LinkArm, LinkActive or LinkActDefer state and receives the link synchronization flow control packet of the peer IB device.

[0058] The above-mentioned device can be implemented by the IB network link layer state synchronization method provided in the first aspect embodiment. For the specific implementation method, please refer to the description in the first aspect embodiment, which will not be repeated here.

[0059] It is understood that the structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0060] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for synchronizing the state of an IB network link layer, characterized in that, include: When the local IB device enters the LinkInitialize state from the LinkDown state, it sends a link synchronization flow control packet to the peer IB device. After receiving the link synchronization flow control packet or initialization flow control packet from the peer IB device, it sends an initialization flow control packet to the peer IB device. The link synchronization flow control packet uses a link layer flow control packet with the OP field set to 4'b1010. When the local IB device's link layer is in the LinkArm, LinkActive, or LinkActDefer state and receives a link synchronization flow control packet from the peer IB device, the local IB device's link layer state machine is returned to the LinkDown state for restart.

2. The IB network link layer state synchronization method according to claim 1, characterized in that, The step of sending an initialization flow control packet to the peer IB device after receiving the link synchronization flow control packet or the initialization flow control packet from the peer IB device further includes: When the peer IB device receives the link synchronization flow control packet from the local IB device, it will return the link layer state machine of the peer IB device to the LinkDown state to restart and enter the LinkInitialize state, and at the same time return the link synchronization flow control packet to the local IB device. When the local IB device receives the link synchronization flow control packet or initialization flow control packet from the peer IB device, it determines that the peer IB device is also in the LinkInitialize state, sends an initialization flow control packet to the peer IB device, and completes the LinkInitialize handshake when it receives the initialization flow control packet returned by the peer device.

3. The IB network link layer state synchronization method according to claim 2, characterized in that, Further includes: After the handshake is completed in the LinkInitialize state, when the IB device receives the CPortState =Arm command from the subnet manager, the link layer state machine controlling the IB device enters the LinkArm state. In the LinkArm state, when the IB device receives the CPortState = Active or message activation command from the subnet manager, it controls the IB device's link layer state machine to enter the LinkActive state.

4. The IB network link layer state synchronization method according to claim 1, characterized in that, The step of returning the link layer state machine of the local IB device to the LinkDown state for restarting further includes: In the LinkArm state, when a link synchronization flow control packet or a physical layer non-establishment link command is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

5. The IB network link layer state synchronization method according to claim 1, characterized in that, The step of returning the link layer state machine of the local IB device to the LinkDown state for restarting further includes: In the LinkActive state, when a link synchronization flow control packet or initialization flow control packet is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

6. The IB network link layer state synchronization method according to claim 1, characterized in that, The step of returning the link layer state machine of the local IB device to the LinkDown state for restarting further includes: In the LinkActDefer state, when a link synchronization flow control packet or a physical layer link not established command is received from the peer IB device and timed out, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

7. An IB network link layer state synchronization device, characterized in that, include: The handshake unit is used to send a link synchronization flow control packet to the peer IB device when the link layer of the local IB device enters the LinkInitialize state from the LinkDown state. After receiving the link synchronization flow control packet or the initialization flow control packet from the peer IB device, it sends an initialization flow control packet to the peer IB device. The link synchronization flow control packet uses a link layer flow control packet with the OP field set to 4'b1010. The restart unit is used to restart the link layer state machine of the local IB device when the link layer of the local IB device is in the LinkArm, LinkActive or LinkActDefer state and receives the link synchronization flow control packet of the peer IB device.

8. The IB network link layer state synchronization device according to claim 7, characterized in that, The handshake unit is further configured to: When the peer IB device receives the link synchronization flow control packet from the local IB device, it will return the link layer state machine of the peer IB device to the LinkDown state to restart and enter the LinkInitialize state, and at the same time return the link synchronization flow control packet to the local IB device. When the local IB device receives the link synchronization flow control packet or initialization flow control packet from the peer IB device, it determines that the peer IB device is also in the LinkInitialize state, sends an initialization flow control packet to the peer IB device, and completes the LinkInitialize handshake when it receives the initialization flow control packet returned by the peer device.

9. The IB network link layer state synchronization device according to claim 8, characterized in that, The handshake unit is further configured to: After the handshake is completed in the LinkInitialize state, when the IB device receives the CPortState =Arm command from the subnet manager, the link layer state machine controlling the IB device enters the LinkArm state. In the LinkArm state, when the IB device receives the CPortState = Active or message activation command from the subnet manager, it controls the IB device's link layer state machine to enter the LinkActive state.

10. The IB network link layer state synchronization device according to claim 7, characterized in that, The restart unit is further used for: In the LinkArm state, when a link synchronization flow control packet or a physical layer non-establishment link command is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

11. The IB network link layer state synchronization device according to claim 7, characterized in that, The restart unit is further used for: In the LinkActive state, when a link synchronization flow control packet or initialization flow control packet is received from the peer IB device, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

12. The IB network link layer state synchronization device according to claim 7, characterized in that, The restart unit is further used for: In the LinkActDefer state, when a link synchronization flow control packet or a physical layer link not established command is received from the peer IB device and timed out, the link layer state machine of the local IB device is returned to the LinkDown state for restart.

Images