Troubleshooting methods and devices

Abstract

This application provides a fault handling method and apparatus. If a fault is detected in a first link directly connected to a second network device, a first FLSN message is sent to a third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link fault based on the first fault flow information. If no FLSN message is received from another directly connected network device, the third network device switches the forwarding path based on the first fault flow information, and the switched forwarding path does not pass through the first network device. Alternatively, when an FLSN message is received from another network device, a second FLSN message is sent to the controller. If a first service node is detected to be unreachable, a third FLSN message is sent to the controller. The third FLSN message includes second fault flow information, so that the controller can determine the first service node is unreachable based on the second fault flow information and send a first notification message to the computing manager.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a fault handling method and apparatus. Background Technology

[0002] In a typical traditional CLOS networking scheme, switches are interconnected to form a large-scale server network.

[0003] When a link between switches fails, the port hardware of the switch detects the change in port status and notifies the internal routing module. The routing module then switches the forwarding path, forwarding service traffic through the backup link.

[0004] The switching time for the aforementioned forwarding paths is typically in the second range. However, in AI scenarios, the desired communication interruption time between switches is in the millisecond range, or even sub-milliseconds. Relying on traditional routing convergence methods will significantly impact the computational efficiency of AI. Moreover, if a service node (e.g., a GPU) within the network becomes unreachable, other switches and management nodes cannot detect this in time. They can only switch forwarding paths again due to routing timeouts or Remote Direct Memory Access (RDMA) connection timeouts, impacting the overall computational efficiency of the network. Summary of the Invention

[0005] In view of this, this application provides a fault handling method and apparatus to solve the problem that the convergence method of existing traditional routing will affect the overall network computing efficiency.

[0006] In a first aspect, this application provides a fault handling method, the method being applied to a first network device, the method comprising:

[0007] If a first link failure directly connected to the second network device is detected, a first Fast Link Status Notification (FLSN) message is sent to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link failure based on the first fault flow information, and switch the forwarding path based on the first fault flow information when it does not receive FLSN messages from other directly connected network devices besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives FLSN messages from other network devices, it sends a second FLSN message to the controller.

[0008] If the first service node to be accessed is detected to be unreachable, a third FLSN message is sent to the controller. The third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable based on the second fault flow information and sends a first notification message to the compute manager.

[0009] Secondly, this application provides a fault handling device, which is applied to a first network device and includes a detection unit and a transmission unit;

[0010] The sending unit is configured to, if the detection unit detects a first link failure directly connected to the second network device, send a first Fast Link Status Notification (FLSN) message to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device determines the first link failure based on the first fault flow information, and switches the forwarding path based on the first fault flow information when it does not receive an FLSN message from any other directly connected network device besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives an FLSN message from another network device, it sends a second FLSN message to the controller.

[0011] The sending unit is further configured to, if it detects that the first service node to be accessed is unreachable, send a third FLSN message to the controller, the third FLSN message including second fault flow information, so that the controller determines that the first service node is unreachable based on the second fault flow information and sends a first notification message to the computing manager.

[0012] Thirdly, this application provides a network device including a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, which in turn cause the processor to perform the method provided in the first aspect of this application.

[0013] Therefore, by applying the fault handling method and apparatus provided in this application, if a fault is detected in the first link directly connected to the second network device, the first network device sends a first Fast Link Status Notification (FLSN) message to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link fault based on the first fault flow information, and switch the forwarding path according to the first fault flow information when it does not receive FLSN messages from other directly connected network devices besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives FLSN messages from other network devices, it sends a second FLSN message to the controller. If the first access service node is detected to be unreachable, the first network device sends a third FLSN message to the controller. The third FLSN message includes second fault flow information, so that the controller can determine the first service node is unreachable based on the second fault flow information and send a first notification message to the computing manager.

[0014] In this way, the control plane and forwarding plane within the network device are managed separately. The control plane retains the original routing protocol, while the forwarding plane uses FLSN messages to announce link failures, enabling rapid detection of link changes. This allows for traffic switching and notification without the service being aware of the issue, significantly reducing the impact of link failures on services. Simultaneously, the network device quickly notifies the controller of service node unreachability information. The controller, in conjunction with the compute controller, rapidly addresses the issue of service node unavailability caused by network failures, further minimizing the impact of network failures on the training and inference processes of large models. Attached Figure Description

[0015] Figure 1 A flowchart of a fault handling method provided in an embodiment of this application;

[0016] Figure 2-A A network topology diagram illustrating a fault scenario provided in an embodiment of this application;

[0017] Figure 2-B A network diagram illustrating another fault scenario provided in this application embodiment;

[0018] Figure 2-C A network topology diagram illustrating another fault scenario provided in this application embodiment;

[0019] Figure 2-D A network topology diagram illustrating another fault scenario provided in this application embodiment;

[0020] Figure 3 This is a structural diagram of the fault handling device provided in the embodiments of this application;

[0021] Figure 4 The network device hardware structure provided in the embodiments of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the corresponding listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] The following is a detailed description of a communication method provided by an embodiment of this application. See also... Figure 1 , Figure 1 This is a flowchart illustrating a fault handling method provided in an embodiment of this application. The method is applied to a first network device, specifically a switching chip included in the first network device. It is understood that the switching chip is a hardware chip that includes a processor, transceiver interface, storage medium, etc.

[0026] The fault handling method provided in this application embodiment may include the following steps.

[0027] Step 110: If a first link failure directly connected to the second network device is detected, a first Fast Link Status Notification (FLSN) message is sent to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link failure based on the first fault flow information, and switch the forwarding path based on the first fault flow information when it does not receive FLSN messages from other directly connected network devices besides the first network device. The switched forwarding path does not pass through the first network device; or, when it receives FLSN messages from other network devices, it sends a second FLSN message to the controller.

[0028] Specifically, the first network device and the second network device are directly connected via a first link, and at least one service node has been connected to the second network device. The first network device and the third network device are also directly connected via a link, and at least one service node has also been connected to the third network device. The second network device and the third network device are also directly connected to other network devices via links.

[0029] The first network device periodically probes the status of the first link. For example, it can periodically probe the port status and port bandwidth utilization of the link to know the status of the first link.

[0030] If the first network device determines that the first link has failed, the first network device generates a first Fast Link Status Notification (FLSN) message, which includes information about the first failed flow. It can be understood that the first FLSN message is a link failure notification message.

[0031] Optionally, the first fault flow information includes the number of service nodes reachable through the first link, the IP address of each service node, and its subnet mask. Further, the first fault flow information may also include the interface identifier of the interface of the first link, which is located within the first network device.

[0032] The first network device sends the first FLSN message to the third network device.

[0033] After receiving the first FLSN message, the third network device obtains the first fault flow information from it.

[0034] Optionally, in this embodiment of the application, the FLSN message includes a Type field, a Number of FLSNs field, an IP Address field, and a Net Mask field.

[0035] The type field indicates that the message is an FLSN message; for example, it can be 16 or a value not used by existing routing protocols. The FLSN information number field, IP address field, and mask field can contain fault flow information. The FLSN information number field carries the number of serving nodes, the IP address field carries the IP address of each serving node, and the mask field carries the mask for each serving node's IP address.

[0036] The type field occupies 8 bits, the FLSN information number field occupies 32 bits, the IP address field occupies 32 bits, and the mask field occupies 32 bits.

[0037] Based on the stored network topology and the first fault flow information, the third network device determines that the first link is faulty. Since the first link is faulty, the third network device determines that the forwarding path via the first network device cannot forward the data packets to the destination service node.

[0038] In this embodiment, network devices establish neighbor relationships and communicate path information to each other. Through the path information, the third network device can know the link connections between other network devices and the service node. Based on the first fault flow information, the third network device determines that the service node cannot be reached through the interface of the first network device. Combined with the network topology, the third network device determines that the first link is faulty.

[0039] At this point, the third network device checks whether it has received an FLSN message from another network device. The third network device can determine whether it has received an FLSN message from another network device (which indicates a direct link failure between the other network device and the second network device) within a preset time period (e.g., 5 seconds).

[0040] If the third network device does not receive an FLSN message from another network device, the third network device switches the forwarding path according to the first fault flow information, so that the switched forwarding path does not pass through the first network device, and the forwarding path passes through other network devices to reach the second network device.

[0041] If the third network device receives FLSN messages from other network devices, it determines, based on the stored network topology, that the links between the other network devices and the second network device, as well as the link between the first and second network devices, are all faulty. The second network device has become an "island," and all service nodes connected to the second network device are unreachable. The third network device aggregates all received FLSN messages and generates a second FLSN message. The third network device sends the second FLSN message to the controller. The controller retrieves fault flow information from the second FLSN message (this fault flow information includes the number of service nodes reachable through the faulty links, the IP address and subnet mask of each service node, and may also include the interface identifier of the interface through which the third network device forwards messages to service nodes). Based on the stored network topology, the controller determines that the second network device has become an "island," and all service nodes connected to the second network device are unreachable. The controller sends a notification message to the compute manager, informing it that the second network device has become an "island," and all service nodes connected to the second network device are unreachable. The compute manager performs fault isolation or checkpointing operations based on the notification message. The compute manager also generates and sends FLSN messages to other service nodes to inform them that service nodes connected to the second network device are unreachable, thus avoiding black hole transmission.

[0042] It should be noted that, during the process of generating the second FLSN message, the third network device can, according to the network topology, aggregate and process FLSN messages sent by multiple network devices that all indicate a link failure reaching the same network device.

[0043] Step 120: If the access service node is detected to be unreachable, a third FLSN message is sent to the controller. The third FLSN message includes second fault flow information, so that the controller determines that the service node is unreachable based on the second fault flow information and sends a first notification message to the computing manager.

[0044] Specifically, at least one service node is also connected to the first network device. The first network device periodically probes the status of the link with each service node, for example, by periodically probing the port status and port bandwidth utilization of the link to obtain the status of the link.

[0045] If the first network device determines that the link between it and the service node is faulty, that is, the service node is unreachable, the first network device generates a third FLSN message, which includes the second fault flow information.

[0046] The first network device sends a third FLSN message to the controller. After receiving the third FLSN message, the controller obtains the second fault flow information from it.

[0047] After receiving the third FLSN message, the controller retrieves the second fault flow information from it. Based on the stored network topology, it determines that one or more service nodes connected to the first network device (the second fault flow information includes the number of unreachable service nodes, their IP addresses, and subnet masks) are unreachable. The controller sends a first notification message to the compute manager, informing it that one or more service nodes connected to the first network device are unreachable. The compute manager performs fault isolation or checkpointing operations based on the first notification message. The compute manager also generates and sends FLSN messages to other service nodes to inform them that one or more service nodes connected to the first network device are unreachable, preventing black-hole transmission.

[0048] It is understandable that the formats of the second and third FLSN messages are the same as those of the first FLSN message, and will not be repeated here.

[0049] In this embodiment, the FLSN message is defined by extending the message header specified in the existing OSPF protocol, and the FLSN message includes fault flow information. Except for the type field, the number of FLSNs field, the IP address field, and the net mask field, all other fields in the message header specified in the existing OSPF protocol are the same as the existing message header fields, and the values ​​of the other fields are set according to the meanings specified in the existing OSPF protocol.

[0050] It should be noted that steps 110 and 120 mentioned above can both be executed by hardware chips included in the first network device, such as a switching chip. The switching chip no longer notifies the upper-layer software of the received FLSN messages; instead, the FLSN message processing (parsing, forwarding, generation), path switching, and other steps are entirely performed by the switching chip's own processor, transceiver interface, and storage medium. The upper-layer software also no longer participates in the processing of FLSN messages or path switching, greatly saving processing time and improving processing efficiency.

[0051] Therefore, by applying the fault handling method provided in this application, if a fault is detected in the first link directly connected to the second network device, the first network device sends a first Fast Link Status Notification (FLSN) message to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link fault based on the first fault flow information, and switch the forwarding path based on the first fault flow information when it does not receive FLSN messages from other directly connected network devices besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives FLSN messages from other network devices, it sends a second FLSN message to the controller. If the first access service node is detected to be unreachable, the first network device sends a third FLSN message to the controller. The third FLSN message includes second fault flow information, so that the controller can determine the first service node is unreachable based on the second fault flow information and send a first notification message to the computing manager.

[0052] In this way, the control plane and forwarding plane within the network device are managed separately. The control plane retains the original routing protocol, while the forwarding plane uses FLSN messages to announce link failures, enabling rapid detection of link changes. This allows for traffic switching and notification without the service being aware of the issue, significantly reducing the impact of link failures on services. Simultaneously, the network device quickly notifies the controller of service node unreachability information. The controller, in conjunction with the compute controller, rapidly addresses the issue of service node unavailability caused by network failures, further minimizing the impact of network failures on the training and inference processes of large models.

[0053] Optionally, in the embodiments of this application, the first network device may also perform various steps of the second network device and the third network device in the above embodiments.

[0054] Specifically, the fourth network device and the fifth network device are directly connected via a third link, and at least one service node has been connected to the fifth network device. The first network device and the fourth network device are also directly connected via a link, and at least one service node has been connected to the first network device. The fifth network device is also directly connected to other network devices via links.

[0055] The fourth network device periodically probes the status of the third link. For example, it can periodically probe the port status and port bandwidth utilization of the link to obtain the status of the third link.

[0056] If the fourth network device determines that the third link is faulty, it generates a fourth FLSN message, which includes information about the third fault flow. The fourth network device then sends the fourth FLSN message to the directly connected first network device.

[0057] After receiving the fourth FLSN message sent by the fourth network device, the first network device obtains the third fault flow information from it.

[0058] Based on the stored network topology and the third fault flow information, the first network device determines that the third link is faulty. Since the third link is faulty, the first network device determines that the forwarding path through the fourth network device cannot forward the data packet to the destination service node.

[0059] At this point, the first network device checks whether it has received an FLSN message from another network device. The first network device can determine whether it has received an FLSN message from another network device (the FLSN message indicates a direct link failure between the other network device and the fifth network device) within a preset time period (e.g., 5 seconds).

[0060] If the first network device does not receive an FLSN message from another network device, the first network device switches the forwarding path according to the third fault flow information so that the switched forwarding path does not pass through the fourth network device, but passes through other network devices to reach the fifth network device.

[0061] If the first network device receives FLSN messages from other network devices, it determines, based on the stored network topology, that the links between the other network devices and the fifth network device, as well as the link between the fourth and fifth network devices, are all faulty. The fifth network device has become an "island," and all service nodes connected to the fifth network device are unreachable. The first network device aggregates all received FLSN messages and generates a fifth FLSN message. The first network device sends the fifth FLSN message to the controller. The controller retrieves the fourth fault flow information from the fifth FLSN message (the fourth fault flow information includes the number of service nodes reachable through the faulty links, the IP address and subnet mask of each service node, and may also include the interface identifier of the interface through which the first network device forwards messages to service nodes). Based on the stored network topology, the controller determines that the fifth network device has become an "island," and all service nodes connected to the fifth network device are unreachable. The controller sends a second notification message to the computing manager, informing the computing manager that the fifth network device has become an "island," and all service nodes connected to the fifth network device are unreachable. Based on the second notification message, the compute manager performs fault isolation or checkpointing operations. The compute manager also generates and sends FLSN messages to other service nodes to inform them that service nodes connected to the fifth network device are unreachable, thus preventing black-hole transmission.

[0062] It should be noted that, during the generation of the fifth FLSN message, the first network device can, according to the network topology, aggregate and process FLSN messages sent by multiple network devices that all indicate a link failure reaching the same network device.

[0063] It is understandable that the formats of the fourth and fifth FLSN messages are the same as those of the first FLSN message, and will not be repeated here.

[0064] Optionally, in this embodiment of the application, after each network device obtains the fault flow information, it will also perform the process of generating a fault forwarding table entry locally based on the fault flow information.

[0065] Specifically, the first network device will be used as an example for explanation.

[0066] After detecting a first link failure, the first network device generates first fault flow information, which includes the number of service nodes reachable through the first link, the IP address of each service node, and the subnet mask.

[0067] The first network device generates a fault forwarding table corresponding to the first fault flow information. The fault forwarding table includes at least one fault forwarding table entry, and each fault forwarding table entry includes a destination address / mask field and an outgoing interface field.

[0068] The destination address / mask field carries the destination address and mask of the service node; the outgoing interface field carries the interface identifier of the interface receiving the fault flow information, or the outgoing interface field carries the interface identifier of the interface of the faulty link.

[0069] In one example, the first fault flow information includes two service nodes: the IP address of service node 1 is xxxx / 24, and the IP address of service node 2 is yyyy / 24. It can be understood that the mask is already reflected in the IP address.

[0070] The first network device generates a first fault forwarding table, which corresponds to the first fault flow information. The first network device generates corresponding fault forwarding table entries for each serving node, as shown in Table 1 below.

[0071] Table 1 First Fault Forwarding Table

[0072] Destination address / mask Output interface xxxx / 24 GE1 / 0 / 1 yyyy / 24 GE1 / 0 / 1

[0073] In Table 1, since the first fault flow information is generated by the first network device, the outgoing interface is specifically the fault link, that is, the interface identifier of the interface included in the first link and the interface is located within the first network device.

[0074] It is understandable that after the third network device receives the first FLSN message, it records the ingress interface that received the first FLSN message and also generates a first fault forwarding table locally corresponding to the first fault flow information. The outgress interface included in the first fault forwarding table is specifically the interface identifier of the interface that received the first FLSN message, that is, the ingress interface of the first FLSN message, and the ingress interface establishes a link with the first network device.

[0075] In this embodiment, after each network device generates or receives each fault flow information, it sequentially generates a corresponding fault forwarding table for each fault flow information according to the generation or reception sequence. Simultaneously, the network device also assigns a sequence number to each fault forwarding table, which can also be used to identify the generation order of the fault forwarding table.

[0076] After generating the first fault forwarding table, the first network device retrieves matching routing table entries and forwarding table entries from its local routing table and local forwarding table based on the destination address / mask field and the outgoing interface field. The first network device then sets all retrieved routing table entries and forwarding table entries to an invalid state (i.e., unavailable).

[0077] Similarly, after other network devices generate the first fault forwarding table, they set all local routing table entries and forwarding table entries that match the destination address / mask field and the outgoing interface field to an invalid state.

[0078] Understandably, when a third network device searches for routing table entries and forwarding table entries, if there are routing table entries and forwarding table entries for the same destination address / mask on different outgoing interfaces, it will set the routing table entries and forwarding table entries that do not match the outgoing interface to a valid state in order to achieve path switching.

[0079] Optionally, in this embodiment of the application, the process of clearing the fault forwarding table by the first network device is also included.

[0080] Specifically, the fault forwarding table is a temporary table and cannot be permanently stored in the network device. The first network device establishes neighbor relationships (e.g., OSPF neighbors, ISIS neighbors) with every other network device in the network. Neighbors communicate path information with each other to generate network topology, update routing tables, and forwarding tables locally.

[0081] The first network device and the sixth network device have established a neighbor relationship. The sixth network device generates and sends path information to the first network device. This path information includes all link information established by each interface of the sixth network device and host routes. The host routes include the destination address of the service node and the outgoing interface, which indicates the interfaces included by the sixth network device.

[0082] The first network device receives path information sent by the sixth network device, records the interface identifier of the incoming interface receiving the path information, and obtains link information and the destination address of the service node from the path information. Based on the link information, the first network device determines whether the link status of each interface established by the sixth network device is normal or faulty. Simultaneously, based on the host routing, the first network device determines whether the service node indicated by the destination address can be reached through the sixth network device.

[0083] After obtaining the destination address and ingress interface identifier of the service node, the first network device checks whether there is a fault forwarding table entry in its local fault forwarding table that matches the destination address. If it does, it checks whether the outgress interface identifier included in the fault forwarding table entry is the same as the ingress interface identifier. If they are the same, the fault forwarding table entry is deleted. If they are different, the path information is collected again, and the above identification process is repeated. If they are still different, the path information collected this time is used, and the fault forwarding table is deleted.

[0084] Simultaneously, based on the destination address, the first network device retrieves a routing table entry and a forwarding table entry matching the destination address from its local routing table and local forwarding table. The interface identifiers of the outgoing and incoming interfaces included in these routing and forwarding table entries are identical. The first network device then deletes these routing and forwarding table entries.

[0085] The fault handling method provided in this application is illustrated below through several scenario examples. For example... Figure 2-A , 2-B As shown in Figures 2-C and 2-D, the network includes devices A, B, C, and D. Devices A and D each connect to two GPUs. Device A is fully connected to both devices B and C, and device D is fully connected to both devices B and C. All fully connected links are equal-cost links.

[0086] exist Figure 2-A In the scenario, GPU3 intends to forward service traffic to GPU1. Initially, after the service traffic sent by GPU3 arrives at device D, device D looks up its local forwarding table based on the destination IP address included in the service traffic and obtains the forwarding table entry that matches the destination IP address. This forwarding table entry may include multiple sub-forwarding table entries. Each sub-forwarding table entry includes an outgoing interface field and a next-hop field. The outgoing interface field indicates the interface ID of the interface included by device D, and the next-hop field indicates the device ID.

[0087] In this embodiment, the forwarding table entry includes two sub-forwarding table entries. Sub-forwarding table entry 1 includes the ID of interface 1, and the next hop is the ID of device B; sub-forwarding table entry 2 includes the ID of interface 2, and the next hop is the ID of device C. Sub-forwarding table entry 2 is set to valid, and sub-forwarding table entry 1 is set to invalid. After the service traffic arrives at device D, device D selects sub-forwarding table entry 2 to forward the service traffic. The forwarding path at this time is GPU3-device D-device C-device A-GPU1.

[0088] If device D detects a link 1 failure between itself and device C at this time, device D will automatically set sub-forwarding table entry 1 to valid and sub-forwarding table entry 2 to invalid. After the service traffic arrives at device D, device D will select sub-forwarding table entry 1 to forward the service traffic. The forwarding path at this time is GPU3-Device D-Device B-Device A-GPU1.

[0089] exist Figure 2-B In the process, GPU1 intends to forward service traffic to GPU3. As mentioned above... Figure 2-A Regarding device D, device A also has a local forwarding table configured, including sub-forwarding table entry 1 (containing the ID of interface 1 and the ID of the next hop B) and sub-forwarding table entry 2 (containing the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table entry 1 to invalid and sub-forwarding table entry 2 to valid. After the service traffic arrives at device A, device A selects sub-forwarding table entry 2 to forward the service traffic. The forwarding path at this time is GPU1-Device A-Device C-Device D-GPU3.

[0090] If device C detects a failure in link 1 between itself and device D at this time, device C generates fault flow information 1. This fault flow information 1 includes the number of service nodes reachable through link 1 (2, GPU3 and GPU4), the IP address of each service node, and its subnet mask. Simultaneously, fault flow information 1 also includes the interface used by device C to reach the service nodes, that is, the interface in device C used to establish link 1.

[0091] Device C generates a fault forwarding table corresponding to fault flow information 1. This fault forwarding table includes two entries, each containing a destination address / mask field and an outgoing interface field. The outgoing interface field indicates the interface in Device C used to establish link 1, as shown in Table 2 below.

[0092] Table 2 Fault Forwarding Table

[0093] Destination address / mask Output interface 1.1.1.1 / 24 (GPU3) GE1 / 0 / 1 1.1.1.2 / 24 (GPU4) GE1 / 0 / 1

[0094] After generating the fault forwarding table, Device C retrieves matching routing table entries and forwarding table entries from its local routing table and local forwarding table based on the destination address / mask field and the outgoing interface field. Device C then sets all retrieved routing table entries and forwarding table entries to an invalid state (i.e., unavailable).

[0095] Device C generates and sends FLSN message 1 to device A. FLSN message 1 includes fault flow information 1.

[0096] After receiving FLSN message 1, device A retrieves fault flow information 1 from it. Based on the stored network topology and fault flow information 1, device A determines that link 1 is faulty.

[0097] Device A generates a fault forwarding table corresponding to fault flow information 1. This fault forwarding table includes two entries, each containing a destination address / mask field and an outgoing interface field. The outgoing interface field indicates the interface in Device A that leads to the service node, as shown in Table 3 below.

[0098] Table 3 Fault Forwarding Table

[0099] Destination address / mask Output interface 1.1.1.1 / 24 (GPU3) GE2 / 0 / 1 1.1.1.2 / 24 (GPU4) GE2 / 0 / 1

[0100] After generating the fault forwarding table, Device A retrieves matching routing table entries and forwarding table entries from its local routing table and local forwarding table based on the destination address / mask field and the outgoing interface field. Device A then sets all retrieved routing table entries and forwarding table entries to an invalid state (for example, setting sub-forwarding table entry 2 to Invalid, i.e., unavailable).

[0101] Understandably, during the process of obtaining routing table entries and forwarding table entries, device A will also obtain sub-forwarding table entry 1. This sub-forwarding table entry 1 can also be used to forward service traffic to GPU3 / GPU4. Device A sets sub-forwarding table entry 1 to be valid. After the service traffic arrives at device A, device A selects sub-forwarding table entry 1 to forward the service traffic. At this time, the forwarding path is GPU1-Device A-Device B-Device D-GPU3.

[0102] exist Figure 2-C In the process, GPU1 intends to forward service traffic to GPU3. As mentioned above... Figure 2-A Regarding device D, device A also has a local forwarding table configured, including sub-forwarding table entry 1 (containing the ID of interface 1 and the ID of the next hop B) and sub-forwarding table entry 2 (containing the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table entry 1 to invalid and sub-forwarding table entry 2 to valid. After the service traffic arrives at device A, device A selects sub-forwarding table entry 2 to forward the service traffic. The forwarding path at this time is GPU1-Device A-Device C-Device D-GPU3.

[0103] If, at this point, device D detects a failure in link 1 between itself and device GPU3, device D determines that GPU3 is unreachable. Device D generates and sends an FLSN message 1 to the controller, which includes fault flow information 1.

[0104] After receiving FLSN message 1, the controller retrieves fault flow information 1 from it. Based on the stored network topology and fault flow information 1, the controller determines that link 1 is faulty. Simultaneously, the controller determines that GPU3 is unreachable.

[0105] After determining that GPU3 is unreachable, the controller generates and sends notification message 1 to the compute manager. This notification message 1 informs the compute manager that GPU3 on access device D is unreachable. Based on notification message 1, the compute manager performs fault isolation or checkpointing operations. The compute manager also generates and sends FLSN message 2 to other service nodes (GPU1, GPU2, GPU4) to inform them that GPU3 on access device D is unreachable, thus preventing black hole transmission.

[0106] Understandably, FLSN message 2 may also include fault flow information 1.

[0107] exist Figure 2-D In the process, GPU1 intends to forward service traffic to GPU3. As mentioned above... Figure 2-A Regarding device D, device A also has a local forwarding table configured, including sub-forwarding table entry 1 (containing the ID of interface 1 and the ID of the next hop B) and sub-forwarding table entry 2 (containing the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table entry 1 to invalid and sub-forwarding table entry 2 to valid. After the service traffic arrives at device A, device A selects sub-forwarding table entry 2 to forward the service traffic. The forwarding path at this time is GPU1-Device A-Device C-Device D-GPU3.

[0108] If at this time, device C detects a failure in link 1 between itself and device D, device C shall proceed as described above. Figure 2-B Device B sends FLSN message 1 to device A in the aforementioned manner. Device B also detects a link 2 failure between itself and device D, and device B also follows the aforementioned procedure. Figure 2-B The FLSN message 2 is sent to device A in the following manner.

[0109] After receiving FLSN message 1 and FLSN message 2, device A proceeds as described above. Figure 2-B In this way, different fault flow information is obtained from them respectively, and multiple fault forwarding tables are generated locally.

[0110] Device A determines, based on the stored network topology and different fault flow information, that both Link 1 between Device C and Device D, and Link 2 between Device B and Device D, are faulty. Device D has become an "island," and GPUs 3 and 4, which are connected to Device D, are unreachable.

[0111] After determining that both GPU3 and GPU4 are unreachable, device A generates and sends an FLSN message 3 to the controller. This FLSN message 3 includes fault flow information 2. It is understandable that since device A can reach GPU3 and GPU4 via multiple equivalent links, fault flow information 2 is a combination of multiple fault information flows. That is, the entries in fault information flow 2 that reach the same service node include different outgoing interface identifiers, as shown in Table 4 below.

[0112] Table 3 Fault Forwarding Table

[0113] Destination address / mask Output interface 1.1.1.1 / 24 (GPU3) GE2 / 0 / 1 1.1.1.1 / 24 (GPU3) GE3 / 0 / 1 1.1.1.2 / 24 (GPU4) GE2 / 0 / 1 1.1.1.2 / 24 (GPU4) GE3 / 0 / 1

[0114] After receiving FLSN message 3, the controller retrieves fault flow information 2 from it. Based on the stored network topology and fault flow information 2, the controller determines that links 1 and 2 are faulty. Simultaneously, the controller determines that GPUs 3 and 4 are unreachable.

[0115] After determining that GPU3 and GPU4 are unreachable, the controller generates and sends notification message 2 to the compute manager. This notification message 2 informs the compute manager that GPU3 and GPU4 on access device D are unreachable. Based on notification message 2, the compute manager performs fault isolation or checkpointing operations. The compute manager also generates and sends FLSN message 4 to other service nodes (GPU1, GPU2) to inform them that GPU3 and GPU4 on access device D are unreachable, thus preventing black-hole transmission.

[0116] Understandably, FLSN message 4 may also include fault flow information 2.

[0117] Based on the same inventive concept, embodiments of this application also provide a fault handling device corresponding to the fault handling method. See also Figure 3 , Figure 3 The fault handling apparatus provided in this application embodiment is applied to a first network device, and the apparatus includes a detection unit 310 and a transmission unit 320.

[0118] The sending unit 320 is configured to send a first Fast Link Status Notification (FLSN) message to a directly connected third network device if the detection unit 310 detects a first link failure directly connected to the second network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link failure based on the first fault flow information, and switch the forwarding path based on the first fault flow information when it does not receive an FLSN message from any other directly connected network device besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives an FLSN message from another network device, it sends a second FLSN message to the controller.

[0119] The sending unit 320 is further configured to, if the detection unit 310 detects that the first service node to be accessed is unreachable, send a third FLSN message to the controller, the third FLSN message including second fault flow information, so that the controller determines that the first service node is unreachable based on the second fault flow information and sends a first notification message to the computing manager.

[0120] Optionally, the device further includes:

[0121] A receiving unit (not shown in the figure) is used to receive a fourth FLSN message sent by a directly connected fourth network device, the fourth FLSN message including third fault flow information;

[0122] A determining unit (not shown in the figure) is used to determine a third link failure based on the third fault flow information, wherein the third link is a direct link between the fourth network device and the fifth network device;

[0123] The switching unit (not shown in the figure) is used to switch the forwarding path according to the third fault flow information if the receiving unit does not receive an FLSN message sent by a directly connected network device other than the fourth network device, so that the switched forwarding path passes through the other network device but not through the fourth network device.

[0124] The sending unit 320 is further configured to, if the receiving unit receives an FLSN message sent by the other network device, send a fifth FLSN message to the controller, the fifth FLSN message including fourth fault flow information, so that the controller determines that the service node accessing the fourth network device is unreachable based on the fourth fault flow information, and sends a second notification message to the computing manager.

[0125] Optionally, the fault flow information includes the number of service nodes reachable through the faulty link, the IP address of each service node, and the subnet mask.

[0126] Optionally, the device further includes:

[0127] The generation unit (not shown in the figure) is used to generate a fault forwarding table corresponding to each fault flow information in sequence when multiple fault flow information is received. Each fault forwarding table includes a destination address / mask field and an outgoing interface field.

[0128] The acquisition unit (not shown in the figure) is used to obtain matching routing table entries and forwarding table entries from the local routing table and the local forwarding table based on the destination address / mask field and the outgoing interface field;

[0129] A setting unit (not shown in the figure) is used to set both the routing table entry and the forwarding table entry to an invalid state;

[0130] The destination address / mask field carries the destination address and mask of the service node; the outgoing interface field carries the interface identifier of the interface receiving the fault flow information, or the outgoing interface field carries the interface identifier of the interface of the faulty link.

[0131] Optionally, the receiving unit (not shown in the figure) is further configured to receive path information sent by the sixth network device, the path information including the destination address of the service node;

[0132] The apparatus further includes a deletion unit (not shown in the figure), configured to delete the fault forwarding table entry and the routing table entry matching the destination address if a fault forwarding table entry matching the destination address exists in the local fault forwarding table and the fault forwarding table entry includes an interface identifier of the interface receiving the path information.

[0133] Therefore, by applying the fault handling apparatus provided in this application, if a fault is detected in the first link directly connected to the second network device, the first network device sends a first Fast Link Status Notification (FLSN) message to the directly connected third network device. The first FLSN message includes first fault flow information, so that the third network device can determine the first link fault based on the first fault flow information, and switch the forwarding path based on the first fault flow information when it does not receive FLSN messages from other directly connected network devices besides the first network device. The switched forwarding path does not pass through the first network device. Alternatively, when it receives FLSN messages from other network devices, it sends a second FLSN message to the controller. If the first access service node is detected to be unreachable, the first network device sends a third FLSN message to the controller. The third FLSN message includes second fault flow information, so that the controller can determine the first service node is unreachable based on the second fault flow information and send a first notification message to the computing manager.

[0134] In this way, the control plane and forwarding plane within the network device are managed separately. The control plane retains the original routing protocol, while the forwarding plane uses FLSN messages to announce link failures, enabling rapid detection of link changes. This allows for traffic switching and notification without the service being aware of the issue, significantly reducing the impact of link failures on services. Simultaneously, the network device quickly notifies the controller of service node unreachability information. The controller, in conjunction with the compute controller, rapidly addresses the issue of service node unavailability caused by network failures, further minimizing the impact of network failures on the training and inference processes of large models.

[0135] Based on the same inventive concept, embodiments of this application also provide a network device, such as... Figure 4 As shown, the system includes a processor 410, a transceiver 420, and a machine-readable storage medium 430. The machine-readable storage medium 430 stores machine-executable instructions that can be executed by the processor 410. The processor 410 is prompted by the machine-executable instructions to execute the fault handling method provided in the embodiments of this application. (The foregoing...) Figure 3 The fault handling device shown can be used as follows: Figure 4 The hardware structure of the network device shown is implemented.

[0136] The aforementioned computer-readable storage medium 430 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the computer-readable storage medium 430 may also be at least one storage device located remotely from the aforementioned processor 410.

[0137] The processor 410 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0138] In this embodiment of the application, the processor 410 reads the machine-executable instructions stored in the machine-readable storage medium 430, and is prompted by the machine-executable instructions to enable the processor 410 itself and the transceiver 420 to execute the fault handling method described in the aforementioned embodiment of the application.

[0139] In addition, this application provides a machine-readable storage medium 430 that stores machine-executable instructions. When called and executed by the processor 410, the machine-executable instructions cause the processor 410 itself and the transceiver 420 to execute the fault handling method described in the aforementioned application.

[0140] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0141] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0142] For the fault handling device and machine-readable storage medium embodiments, since the methods involved are basically similar to those in the aforementioned method embodiments, the description is relatively simple, and relevant details can be found in the descriptions of the method embodiments.

[0143] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A failure handling method characterized by, The method is applied to a first network device, and the method comprises: if a first link directly connected with a second network device is detected to be faulty, a first fast link state notification (FLSN) message is sent to a third network device directly connected, the first FLSN message comprising first fault flow information, so that the third network device determines the first link to be faulty according to the first fault flow information, and switches a forwarding path according to the first fault flow information when no FLSN message sent by other network devices directly connected except the first network device is received, the switched forwarding path not passing through the first network device; or, when the FLSN message sent by other network devices is received, a second FLSN message is sent to a controller; if an accessed service node is detected to be unreachable, a third FLSN message is sent to the controller, the third FLSN message comprising second fault flow information, so that the controller determines the service node to be unreachable according to the second fault flow information, and sends a first notification message to a computing manager.

2. The method of claim 1, wherein, The method further comprises: receiving a fourth FLSN message sent by a fourth network device directly connected, the fourth FLSN message comprising third fault flow information; determining a third link to be faulty according to the third fault flow information, the third link being a directly connected link between the fourth network device and a fifth network device; if no FLSN message sent by other network devices directly connected except the fourth network device is received, switching a forwarding path according to the third fault flow information, so that the switched forwarding path passes through the other network devices and does not pass through the fourth network device; if a fifth FLSN message sent by the other network devices is received, sending a fifth FLSN message to the controller, the fifth FLSN message comprising fourth fault flow information, so that the controller determines a service node accessed by the fifth network device to be unreachable according to the fourth fault flow information, and sends a second notification message to the computing manager.

3. The method according to any of claims 1 or 2, characterized in that, The fault flow information comprises a number of service nodes reachable through a faulty link, an IP address of each service node, and a mask.

4. The method of claim 3, wherein, The method further comprises: when a plurality of fault flow information is received, generating a fault forwarding table corresponding to each fault flow information in sequence, each fault forwarding table comprising a destination address / mask field and an out-interface field; obtaining matched routing table entries and forwarding table entries from a local routing table and a local forwarding table according to the destination address / mask field and the out-interface field; setting the routing table entries and the forwarding table entries to be in an invalid state; wherein the destination address / mask field carries a destination address and a mask of the service node, and the out-interface field carries an interface identifier of an interface receiving the fault flow information, or the out-interface field carries an interface identifier of an interface of a faulty link.

5. The method of claim 4, wherein, The method further comprises: receiving path information sent by a sixth network device, the path information comprising a destination address of a service node; If a fault forwarding table entry matching the destination address exists in the local fault forwarding table and the fault forwarding table entry includes an interface identifier of the interface receiving the path information, the fault forwarding table entry is deleted, and a routing table entry and a forwarding table entry matching the destination address are deleted.

6. A failure handling apparatus characterized by comprising: The device is applied to a first network device, and the device comprises a detection unit and a sending unit; The sending unit is configured to, if the detection unit detects a first link fault directly connected with a second network device, send a first fast link state notification (FLSN) message to a third network device directly connected, the first FLSN message comprising first fault flow information, so that the third network device determines the first link fault according to the first fault flow information, and switches a forwarding path according to the first fault flow information when no FLSN message sent by other network devices directly connected except the first network device is received, the switched forwarding path not passing through the first network device; or, when a FLSN message sent by other network devices is received, sends a second FLSN message to a controller. The sending unit is further configured to, if the detection unit detects that a first service node accessed is unreachable, send a third FLSN message to the controller, the third FLSN message comprising second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to a computing manager.

7. The apparatus of claim 6, wherein, The device further comprises: A receiving unit configured to receive a fourth FLSN message sent by a fourth network device directly connected, the fourth FLSN message comprising third fault flow information; A determination unit configured to determine a third link fault according to the third fault flow information, the third link being a direct link between the fourth network device and a fifth network device; A switching unit configured to, if the receiving unit does not receive a FLSN message sent by other network devices directly connected except the fourth network device, switch a forwarding path according to the third fault flow information, so that the switched forwarding path passes through the other network devices and does not pass through the fourth network device. The sending unit is further configured to, if the receiving unit receives a fifth FLSN message sent by the other network devices, send a fifth FLSN message to the controller, the fifth FLSN message comprising fourth fault flow information, so that the controller determines that a service node accessed by the fifth network device is unreachable according to the fourth fault flow information, and sends a second notification message to the computing manager.

8. The apparatus of claim 7, wherein, The fault flow information comprises a number of service nodes reachable through a fault link, an IP address of each service node, and a mask.

9. The apparatus of claim 8, wherein, The device further comprises: A generation unit configured to, when a plurality of fault flow information is received, generate a fault forwarding table corresponding to each fault flow information in sequence, each fault forwarding table comprising a destination address / mask field and an out-interface field. The acquisition unit is configured to acquire matched routing table entries and forwarding table entries from a local routing table and a local forwarding table according to the destination address / mask field and the out-interface field; The setting unit is configured to set the routing table entries and the forwarding table entries to be invalid. The destination address / mask field carries a destination address and a mask of the service node, and the out-interface field carries an interface identifier of an interface receiving the fault flow information, or the out-interface field carries an interface identifier of an interface of a fault link.

10. The apparatus of claim 9, wherein, The receiving unit is further configured to receive path information sent by the sixth network device, the path information including a destination address of a service node. The device further includes a deletion unit configured to, if there is a fault forwarding table entry matching the destination address in a local fault forwarding table and the fault forwarding table entry includes an interface identifier of an interface receiving the path information, delete the fault forwarding table entry and delete routing table entries and forwarding table entries matching the destination address.

Images