Fault processing method and device

By using fast link status notification messages in network devices to achieve fast awareness and forwarding path switching of link failures, the communication interruption and unreachable service nodes caused by traditional routing convergence methods are solved, and AI computing efficiency and network failures are improved Processing capability.

CN118945105AActive Publication Date: 2024-11-12NEW H3C TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410719455.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-04
Publication Date
2024-11-12
Estimated Expiration
2044-06-04

Smart Images

  • Figure CN118945105A_ABST
    Figure CN118945105A_ABST
Patent Text Reader

Abstract

The invention provides a fault processing method and device, and the method comprises the steps: transmitting a first FLSN message to a third network device if a fault of a first link directly connected with a second network device is detected, and enabling the third network device to determine the fault of the first link according to the first fault flow information, when an FLSN message sent by other directly connected network equipment is not received, a forwarding path is switched according to the first fault flow information, and the switched forwarding path does not pass through the first network equipment; or, when the FLSN message sent by other network equipment is received, a second FLSN message is sent to the controller; if it is detected that the first service node is not reachable, a third FLSN message is sent to the controller, and the third FLSN message comprises second fault flow information, so that the controller determines that the first service node is not reachable according to the second fault flow information and sends a first notification message to a computing manager.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a fault handling method and device. Background Art

[0002] In the traditional typical CLOS networking solution, switches are interconnected to form a large-scale server network.

[0003] When a link between switches fails, the port hardware included in the switch senses the port status change and notifies the internal routing module. The routing module switches the forwarding path and forwards the service traffic through the backup link.

[0004] The switching time of the above forwarding path is usually in seconds. However, in the AI ​​scenario, it is expected that the communication interruption time between switches is required to be in milliseconds or even sub-milliseconds. Relying on the convergence method of traditional routing will greatly affect the computing efficiency of AI. Moreover, if a service node (for example, GPU) in the network is unreachable, other switches and management nodes cannot perceive it in time, and can only re-switch the forwarding path through routing timeout or remote direct memory access (English: Remote Direct Memory Access, referred to as: RDMA) connection timeout, affecting the computing efficiency of the entire network. Summary of the invention

[0005] In view of this, the present application provides a fault handling method and device to solve the problem that the convergence mode of existing traditional routing will affect the efficiency of the entire network calculation.

[0006] In a first aspect, the present application provides a fault handling method, which is applied to a first network device, and the method includes:

[0007] If a first link failure directly connected to the second network device is detected, a first fast link state notification FLSN message is sent to a directly connected third network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when no FLSN message sent by other directly connected network devices other than the first network device is received, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, a second FLSN message is sent to the controller;

[0008] If it is detected that the first service node accessed is unreachable, a third FLSN message is sent to the controller, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0009] In a second aspect, the present application provides a fault handling device, the device is applied to a first network device, the device includes a detection unit and a sending unit;

[0010] The sending unit is used for sending a first fast link state notification FLSN message to a directly connected third network device if the detection unit detects a first link failure directly connected to the second network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the forwarding path after switching does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller;

[0011] The sending unit is also used to send a third FLSN message to the controller if it is detected that the first service node accessed is unreachable, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0012] In a third aspect, the present application provides a network device, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to execute the method provided in the first aspect of the present application.

[0013] Therefore, by applying the fault handling method and device provided in the present application, if a first link failure directly connected to a second network device is detected, the first network device sends a first fast link state notification FLSN message to a directly connected third network device, and the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller; if it is detected that the first service node accessed is unreachable, the first network device sends a third FLSN message to the controller, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0014] In this way, the control plane and forwarding plane in the network device are managed separately. The control plane continues to use the original routing protocol, and the forwarding plane uses FLSN messages to implement link fault notifications to quickly perceive link changes. In this way, traffic switching and notification are achieved without the business being aware of it, greatly reducing the impact of link failures on the business. At the same time, the network device also quickly notifies the controller of the service node unreachable information, and the controller and the computing controller work together to quickly handle the problem of service node unavailability caused by network failures, which also greatly reduces the impact of network failures on large model training and reasoning processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a fault handling method provided in an embodiment of the present application;

[0016] Figure 2-A A network diagram of a fault scenario provided in an embodiment of the present application;

[0017] Figure 2-B A network diagram of another fault scenario provided in an embodiment of the present application;

[0018] Figure 2-C A network diagram of another fault scenario provided in an embodiment of the present application;

[0019] Figure 2-D A network diagram of another fault scenario provided in an embodiment of the present application;

[0020] Figure 3 A structural diagram of a fault handling device provided in an embodiment of the present application;

[0021] Figure 4 The network device hardware structure provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more corresponding listed items.

[0024] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] A communication method provided in an embodiment of the present application is described in detail below. Figure 1 , Figure 1 A flowchart of a fault handling method provided in an embodiment of the present application. The method is applied to a first network device, specifically a switching chip included in the first network device. It can be understood that the switching chip is a hardware chip, which includes a processor, a transceiver interface, a storage medium, etc.

[0026] The fault handling method provided in the embodiment of the present application may include the following steps.

[0027] Step 110: If a first link failure directly connected to the second network device is detected, a first fast link state notification FLSN message is sent to a directly connected third network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when no FLSN message is received from other directly connected network devices except the first network device, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, a second FLSN message is sent to the controller;

[0028] Specifically, the first network device is directly connected to the second network device through a first link, and at least one service node is connected to the second network device. The first network device is also directly connected to the third network device through a link, and at least one service node is also connected to the third network device. The second network device and the third network device are also directly connected to other network devices through links.

[0029] The first network device periodically detects the state of the first link. For example, it may periodically detect the port state of the link, the occupancy rate of the port bandwidth, etc. to obtain the state of the first link.

[0030] If the first network device determines that the first link fails, the first network device generates a first Fast Link Status Notification (FLSN) message, and the first FLSN message includes first fault flow information. It can be understood that the first FLSN message is a link failure notification message.

[0031] Optionally, the first fault flow information includes the number of service nodes reachable through the first link, the IP address and mask of each service node. Further, the first fault flow information may also include an interface identifier of an interface of the first link, the interface being located in the first network device.

[0032] The first network device sends a first FLSN message to the third network device.

[0033] After receiving the first FLSN message, the third network device obtains the first fault flow information therefrom.

[0034] Optionally, in an embodiment of the present application, the FLSN message includes a type (Type) field, a FLSN information number (Number of FLSNs) field, an IP address (IP Address) field, and a mask (Net Mask) field.

[0035] The value of the type field is used to indicate that the message is a FLSN message, for example, 16 or a value not used by an existing protocol routing protocol, and the FLSN information number field, the IP address field, and the mask field can be fault flow information. The FLSN information number field is used to carry the number of service nodes, the IP address field is used to carry the IP address of each service node, and the mask field is used to carry the mask of the IP address of each service node.

[0036] The type field occupies 8 bits, the FLSN information number field occupies 32 bits, the IP address field occupies 32 bits, and the mask field occupies 32 bits.

[0037] According to the stored network topology and the first fault flow information, the third network device determines that the first link is faulty. Since the first link is faulty, the third network device determines that the forwarding path through the first network device cannot forward the data message to the destination service node.

[0038] In the embodiment of the present application, the network devices establish a neighbor relationship and notify each other of path information. Through the path information, the third network device can know other network devices and the link connection relationship between other network devices and the service node. Through the first fault flow information, the third network device determines that the service node cannot be reached through the interface of the first network device, and then combined with the network topology, the third network device determines that the first link is faulty.

[0039] At this time, the third network device identifies whether it has received the FLSN message sent by other network devices. The third network device can determine whether it has received the FLSN message sent by other network devices within a preset time period (for example, 5s) (the FLSN message indicates a direct link failure between the other network devices and the second network device).

[0040] If the third network device does not receive the FLSN message sent by other network devices, the third network device switches the forwarding path according to the first fault flow information so that the switched forwarding path does not pass through the first network device, and the forwarding path passes through other network devices to reach the second network device.

[0041] If the third network device receives the FLSN message sent by other network devices, the third network device determines, based on the stored network topology, that the links between the other network devices and the second network device and the links between the first network device and the second network device are all faulty, that the second network device has become an "island", and that the service nodes connected to the second network device are all unreachable. The third network device aggregates and processes all the received FLSN messages to generate a second FLSN message. The third network device sends the second FLSN message to the controller. The controller obtains the fault flow information from the second FLSN message (the fault flow information includes the number of service nodes reachable through the faulty link, the IP address and mask of each service node, and the fault flow information may also include the interface identifier of the interface through which the third network device forwards the message to the service node), and determines, based on the stored network topology, that the second network device has become an "island", and that the service nodes connected to the second network device are all unreachable. The controller sends a notification message to the computing manager, which is used to inform the computing manager that the second network device has become an "island", and that the service nodes connected to the second network device are all unreachable. The computing manager performs fault isolation or checkpoint-related operations based on the notification message. The computing manager also generates and sends FLSN messages to other service nodes to inform other service nodes that the service nodes connected to the second network device are unreachable to avoid black hole transmission.

[0042] It should be noted that, when generating the second FLSN message, the third network device may, according to the network topology, aggregate and process the FLSN messages sent by multiple network devices and all indicating link failures reaching the same network device.

[0043] Step 120: If it is detected that the accessed service node is unreachable, a third FLSN message is sent to the controller, wherein the third FLSN message includes second fault flow information, so that the controller determines that the service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0044] Specifically, the first network device is also connected to at least one service node. The first network device periodically detects the state of the link between each service node, for example, the port state of the link, the occupancy rate of the port bandwidth, etc. can be periodically detected to learn the link state.

[0045] If the first network device determines that the link between the first network device and the service node is faulty, that is, the service node is unreachable, the first network device generates a third FLSN message, and the third FLSN message includes the second fault flow information.

[0046] The first network device sends a third FLSN message to the controller. After receiving the third FLSN message, the controller obtains the second fault flow information therefrom.

[0047] After receiving the third FLSN message, the controller obtains the second fault flow information from the third FLSN message. According to the stored network topology, it is determined that one or more service nodes connected to the first network device (the second fault flow information includes the number of unreachable service nodes, IP addresses and masks) are unreachable. The controller sends a first notification message to the computing manager, and the first notification message is used to inform the computing manager that one or more service nodes connected to the first network device are unreachable. The computing manager performs fault isolation or checkpoint related operations based on the first notification message. The computing manager also generates and sends FLSN messages to other service nodes to inform other service nodes that one or more service nodes connected to the first network device cannot be reached to avoid black hole transmission.

[0048] It can be understood that the formats of the second FLSN message and the third FLSN message are the same as the format of the first FLSN message, which will not be repeated here.

[0049] In the embodiment of the present application, the FLSN message is defined by extending the message header specified in the existing OSPF protocol, and the FLSN message includes fault flow information. Except for the type field, the number of FLSNs field, the IP address field, and the Net Mask field, the other fields of the message header specified in the existing OSPF protocol are the same as the existing message header fields, and the values ​​of the other fields are set according to the meanings specified in the existing OSPF protocol.

[0050] It should be noted that the aforementioned steps 110 and 120 can be performed by a hardware chip included in the first network device, for example, a switching chip. The switching chip no longer notifies the upper-layer software of the received FLSN message, and the FLSN message processing (parsing, forwarding, generation), path switching and other steps are completely performed by the processor, transceiver interface, and storage medium of the switching chip itself. The upper-layer software also does not participate in the processing of the FLSN message, path switching and other steps, which greatly saves processing time and improves processing efficiency.

[0051] Therefore, by applying the fault handling method provided in the present application, if a first link failure directly connected to a second network device is detected, the first network device sends a first fast link state notification FLSN message to a directly connected third network device, and the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller; if it is detected that the first service node accessed is unreachable, the first network device sends a third FLSN message to the controller, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0052] In this way, the control plane and forwarding plane in the network device are managed separately. The control plane continues to use the original routing protocol, and the forwarding plane uses FLSN messages to implement link fault notifications to quickly perceive link changes. In this way, traffic switching and notification are achieved without the business being aware of it, greatly reducing the impact of link failures on the business. At the same time, the network device also quickly notifies the controller of the service node unreachable information, and the controller and the computing controller work together to quickly handle the problem of service node unavailability caused by network failures, which also greatly reduces the impact of network failures on large model training and reasoning processes.

[0053] Optionally, in the embodiment of the present application, the first network device may also execute various steps of the second network device and the third network device in the above embodiment.

[0054] Specifically, the fourth network device is directly connected to the fifth network device via a third link, and at least one service node is connected to the fifth network device. The first network device is also directly connected to the fourth network device via a link, and at least one service node is also connected to the first network device. The fifth network device is also directly connected to other network devices via links.

[0055] The fourth network device periodically detects the state of the third link. For example, it may periodically detect the port state of the link, the occupancy rate of the port bandwidth, etc. to acquire the state of the third link.

[0056] If the fourth network device determines that the third link fails, the fourth network device generates a fourth FLSN message, the fourth FLSN message including the third fault flow information, and sends the fourth FLSN message to the directly connected first network device.

[0057] After receiving the fourth FLSN message sent by the fourth network device, the first network device obtains the third fault flow information therefrom.

[0058] According to the stored network topology and the third fault flow information, the first network device determines that the third link fails. Since the third link has failed, the first network device determines that the forwarding path through the fourth network device cannot forward the data message to the destination service node.

[0059] At this time, the first network device identifies whether it has received the FLSN message sent by other network devices. The first network device can determine whether it has received the FLSN message sent by other network devices within a preset time period (for example, 5s) (the FLSN message indicates a direct link failure between the other network devices and the fifth network device).

[0060] If the first network device does not receive the FLSN message sent by other network devices, the first network device switches the forwarding path according to the third fault flow information so that the switched forwarding path does not pass through the fourth network device, and the forwarding path passes through other network devices to reach the fifth network device.

[0061] If the first network device receives the FLSN message sent by other network devices, the first network device determines, based on the stored network topology, that the links between the other network devices and the fifth network device and the links between the fourth network device and the fifth network device are all faulty, that the fifth network device has become an "island", and that the service nodes connected to the fifth network device are all unreachable. The first network device aggregates and processes all the received FLSN messages to generate a fifth FLSN message. The first network device sends the fifth FLSN message to the controller. The controller obtains the fourth fault flow information from the fifth FLSN message (the fourth fault flow information includes the number of service nodes reachable through the faulty link, the IP address and mask of each service node, and the fourth fault flow information may also include the interface identifier of the interface through which the first network device forwards the message to the service node), and determines, based on the stored network topology, that the fifth network device has become an "island", and that the service nodes connected to the fifth network device are all unreachable. The controller sends a second notification message to the computing manager, and the second notification message is used to inform the computing manager that the fifth network device has become an "island", and that the service nodes connected to the fifth network device are all unreachable. The computing manager performs fault isolation or checkpoint-related operations according to the second notification message. The computing manager also generates and sends FLSN messages to other service nodes to inform other service nodes that the service nodes connected to the fifth network device are unreachable to avoid black hole transmission.

[0062] It should be noted that, when generating the fifth FLSN message, the first network device may, according to the network topology, aggregate and process the FLSN messages sent by multiple network devices and all indicating link failures reaching the same network device.

[0063] It can be understood that the formats of the fourth FLSN message and the fifth FLSN message are the same as the format of the first FLSN message, which will not be repeated here.

[0064] Optionally, in the embodiment of the present application, after each network device obtains the fault flow information, it will also execute a process of generating a fault forwarding table entry locally according to the fault flow information.

[0065] Specifically, the first network device is taken as an example for description.

[0066] After the first network device detects a first link failure, it generates first fault flow information, where the first fault flow information includes the number of service nodes reachable through the first link, and the IP address and mask of each service node.

[0067] The first network device generates a fault forwarding table corresponding to the first fault flow information. The fault forwarding table includes at least one fault forwarding table entry, and each fault forwarding table entry includes a destination address / mask field and an outbound interface field.

[0068] The destination address / mask field carries the destination address and mask of the service node; the outbound interface field carries the interface identifier of the interface receiving the fault flow information, or the outbound interface field carries the interface identifier of the interface of the faulty link.

[0069] In an example, the first fault flow information includes 2 service nodes, the IP address of service node 1 is xxxx / 24; the IP address of service node 2 is yyyy / 24. It can be understood that the mask is reflected in the IP address.

[0070] The first network device generates a first fault forwarding table, which corresponds to the first fault flow information. The first network device generates a corresponding fault forwarding table entry for each service node, as shown in Table 1 below.

[0071] Table 1 The first fault forwarding table

[0072] Destination address / mask Outbound interface xxxx / 24 GE1 / 0 / 1 yyyy / 24 GE1 / 0 / 1

[0073] In Table 1, since the first fault flow information is generated by the first network device, the outgoing interface is specifically the faulty link, that is, the interface identifier of the interface included in the first link and the interface is in the first network device.

[0074] It can be understood that after the third network device receives the first FLSN message, it records the input interface for receiving the first FLSN message, and also locally generates a first fault forwarding table corresponding to the first fault flow information. The output interface included in the first fault forwarding table is specifically the interface identifier of the interface for receiving the first FLSN message, that is, the input interface of the first FLSN message, and the input interface establishes a link with the first network device.

[0075] In the embodiment of the present application, after each network device generates or receives each fault flow information, it generates a corresponding fault forwarding table for each fault flow information in sequence according to the timing of generation or reception. At the same time, the network device also sets a sequence number for each fault forwarding table, and the sequence number can also be used to identify the generation order of the fault forwarding table.

[0076] After the first network device generates the first fault forwarding table, it obtains matching routing table items and forwarding table items from the local routing table and the local forwarding table according to the destination address / mask field and the outgoing interface field. The first network device sets the obtained routing table items and forwarding table items to invalid state (i.e., unavailable).

[0077] Similarly, after generating the first fault forwarding table, other network devices set the local routing table entries and forwarding table entries that match the destination address / mask field and the outbound interface field to an invalid state.

[0078] It is understandable that when the third network device is searching for routing table entries and forwarding table entries, if there are routing table entries and forwarding table entries with different output interfaces for the same destination address / mask, the routing table entries and forwarding table entries with unmatched output interfaces will be set to a valid state to achieve path switching.

[0079] Optionally, in the embodiment of the present application, the process of the first network device clearing the fault forwarding table is also included.

[0080] Specifically, the fault forwarding table is a temporary table and cannot be permanently stored in the network device. The first network device establishes a neighbor (e.g., OSPF neighbor, ISIS neighbor) relationship with each network device in the network. Neighbors mutually announce path information to generate network topology, update routing tables, and forwarding tables locally.

[0081] A neighbor relationship has been established between the first network device and the sixth network device. The sixth network device generates and sends path information to the first network device, the path information including all link information established by each interface of the sixth network device and a host route. The host route includes a destination address of a service node and an outgoing interface, the outgoing interface indicating an interface included in the sixth network device.

[0082] The first network device receives the path information sent by the sixth network device, records the interface identifier of the input interface of the received path information, and obtains the link information and the destination address of the service node from the path information. Based on the link information, the first network device determines whether the link status of the link established by each interface of the sixth network device is a normal state or a fault state. At the same time, based on the host route, the first network device determines whether the service node indicated by the destination address can be reached through the sixth network device.

[0083] After the first network device obtains the destination address of the service node and the interface identifier of the incoming interface, it identifies whether there is a fault forwarding table entry matching the destination address in the local fault forwarding table. If so, it identifies whether the interface identifier of the outgoing interface included in the fault forwarding table entry is the same as the interface identifier of the incoming interface. If they are the same, the fault forwarding table entry is deleted. If they are different, the path information is collected again and the above identification process is repeated. If they are still different, the path information collected this time shall prevail and the fault forwarding table shall be deleted.

[0084] At the same time, according to the destination address, the first network device obtains a routing table item and a forwarding table item that match the destination address from a local routing table and a local forwarding table. The interface identifier of the outgoing interface included in the routing table item and the forwarding table item is the same as the interface identifier of the incoming interface. The first network device deletes the routing table item and the forwarding table item.

[0085] The following describes the troubleshooting method provided by this application through multiple scenario examples. Figure 2-A , 2-B As shown in 2-C and 2-D, the network includes device A, device B, device C and device D. Device A and device D are connected to two GPUs respectively. Device A is fully connected with device B and device C respectively, and device D is fully connected with device B and device C respectively. All fully connected links are equivalent links.

[0086] exist Figure 2-A In the example, GPU3 intends to forward service traffic to GPU1. In the initial state, after the service traffic sent by GPU3 reaches device D, device D searches the local forwarding table according to the destination IP address included in the service traffic, and obtains a forwarding table entry that matches the destination IP address. The forwarding table entry may include multiple sub-forwarding table entries, each of which includes an outgoing interface field and a next hop field. The outgoing interface field indicates the interface ID of the interface included in device D, and the next hop field indicates the device ID.

[0087] In the embodiment of the present application, the forwarding table entry includes two sub-forwarding table entries, sub-forwarding table entry 1 includes the ID of interface 1, and the next hop is the ID of device B; sub-forwarding table entry 2 includes the ID of interface 2, and the next hop is the ID of device C. Among them, sub-forwarding table entry 2 is set to valid, and sub-forwarding table entry 1 is set to invalid. After the service traffic reaches device D, device D selects sub-forwarding table entry 2 to forward the service traffic. At this time, the forwarding path is GPU3-device D-device C-device A-GPU1.

[0088] If at this time, device D detects that link 1 between itself and device C fails, device D automatically sets sub-forwarding table item 1 in the local forwarding table to valid and sub-forwarding table item 2 to invalid. After the service traffic reaches device D, device D selects sub-forwarding table item 1 to forward the service traffic. At this time, the forwarding path is GPU3-device D-device B-device A-GPU1.

[0089] exist Figure 2-B In the example above, GPU1 wants to forward service traffic to GPU3. Figure 2-A As described in the description of device D, a local forwarding table has also been configured in device A, and includes sub-forwarding table item 1 (including the ID of interface 1 and the ID of the next hop B) and sub-forwarding table item 2 (including the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table item 1 to invalid and sub-forwarding table item 2 to valid. After the service traffic reaches device A, device A selects sub-forwarding table item 2 to forward the service traffic. At this time, the forwarding path is GPU1-device A-device C-device D-GPU3.

[0090] If at this time, device C detects that link 1 between itself and device D fails, device C generates fault flow information 1, which includes the number of service nodes reachable through link 1 (2, GPU3, GPU4), the IP address and mask of each service node. At the same time, fault flow information 1 also includes the interface through which device C reaches the service node, that is, the interface in device C used to establish link 1.

[0091] Device C generates a fault forwarding table corresponding to fault flow information 1. The fault forwarding table includes two fault forwarding table entries, each of which includes a destination address / mask field and an outbound interface field. The outbound interface field indicates the interface in device C used to establish link 1. This is shown in Table 2 below.

[0092] Table 2 Fault forwarding table

[0093] Destination address / mask Outbound interface 1.1.1.1 / 24(GPU3) GE1 / 0 / 1 1.1.1.2 / 24(GPU4) GE1 / 0 / 1

[0094] After device C generates the fault forwarding table, it obtains matching routing table items and forwarding table items from the local routing table and the local forwarding table according to the destination address / mask field and the outgoing interface field. Device C sets the obtained routing table items and forwarding table items to invalid state (Invalid, i.e., unavailable).

[0095] Device C generates and sends FLSN message 1 to device A. The FLSN message 1 includes fault flow information 1.

[0096] After receiving FLSN message 1, device A obtains fault flow information 1 from it. According to the stored network topology and fault flow information 1, device A determines that link 1 is faulty.

[0097] Device A generates a fault forwarding table corresponding to fault flow information 1. The fault forwarding table includes two fault forwarding table entries, each of which includes a destination address / mask field and an outbound interface field. The outbound interface field indicates the interface in device A that reaches the service node. See Table 3 below.

[0098] Table 3 Fault forwarding table

[0099] Destination address / mask Outbound interface 1.1.1.1 / 24(GPU3) GE2 / 0 / 1 1.1.1.2 / 24(GPU4) GE2 / 0 / 1

[0100] After device A generates the fault forwarding table, it obtains matching routing table items and forwarding table items from the local routing table and the local forwarding table according to the destination address / mask field and the outgoing interface field. Device A sets the obtained routing table items and forwarding table items to an invalid state (for example, setting sub-forwarding table item 2 to Invalid, i.e., unavailable).

[0101] It is understandable that, in the process of obtaining the routing table items and the forwarding table items, device A will also obtain the sub-forwarding table item 1, which can also be used to forward the service traffic to GPU3 / GPU4. Device A sets the sub-forwarding table item 1 to be valid. After the service traffic reaches device A, device A selects the sub-forwarding table item 1 to forward the service traffic. At this time, the forwarding path is GPU1-device A-device B-device D-GPU3.

[0102] exist Figure 2-C In the example above, GPU1 wants to forward service traffic to GPU3. Figure 2-A As described in the description of device D, a local forwarding table has also been configured in device A, and includes sub-forwarding table item 1 (including the ID of interface 1 and the ID of the next hop B) and sub-forwarding table item 2 (including the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table item 1 to invalid and sub-forwarding table item 2 to valid. After the service traffic reaches device A, device A selects sub-forwarding table item 2 to forward the service traffic. At this time, the forwarding path is GPU1-device A-device C-device D-GPU3.

[0103] If at this time, device D detects that link 1 between itself and device GPU3 is faulty, device D determines that GPU3 is unreachable. Device D generates and sends FLSN message 1 to the controller, and the FLSN message 1 includes fault flow information 1.

[0104] After receiving FLSN message 1, the controller obtains fault flow information 1 from it. According to the stored network topology and fault flow information 1, the controller determines that link 1 is faulty. At the same time, the controller determines that GPU 3 is unreachable.

[0105] After the controller determines that GPU3 is unreachable, it generates and sends a notification message 1 to the computing manager, which is used to inform the computing manager that GPU3 connected to device D is unreachable. The computing manager performs fault isolation or checkpoint-related operations based on notification message 1. The computing manager also generates and sends FLSN message 2 to other service nodes (GPU1, GPU2, GPU4) to inform other service nodes that GPU3 connected to device D is unreachable to avoid black hole transmission.

[0106] It can be understood that the FLSN message 2 may also include the fault flow information 1 .

[0107] exist Figure 2-D In the example above, GPU1 wants to forward service traffic to GPU3. Figure 2-A As described in the description of device D, a local forwarding table has also been configured in device A, and includes sub-forwarding table item 1 (including the ID of interface 1 and the ID of the next hop B) and sub-forwarding table item 2 (including the ID of interface 2 and the ID of the next hop C). Device A sets sub-forwarding table item 1 to invalid and sub-forwarding table item 2 to valid. After the service traffic reaches device A, device A selects sub-forwarding table item 2 to forward the service traffic. At this time, the forwarding path is GPU1-device A-device C-device D-GPU3.

[0108] If device C detects that link 1 between itself and device D is faulty, device C follows the above Figure 2-B Device B also detects the failure of link 2 between itself and device D, and device B also follows the above Figure 2-B FLSN message 2 is sent to device A in the same manner as above.

[0109] After receiving FLSN message 1 and FLSN message 2, device A follows the above Figure 2-B In this way, different fault flow information is obtained respectively, and multiple fault forwarding tables are generated locally.

[0110] Device A determines through the stored network topology and different fault flow information that link 1 between device C and device D and link 2 between device B and device D are both faulty, and device D has become an "island". GPU3 and GPU4 connected to device D are unreachable.

[0111] After device A determines that both GPU3 and GPU4 are unreachable, it generates and sends FLSN message 3 to the controller, and the FLSN message 3 includes fault flow information 2. It can be understood that since device A can reach GPU3 and GPU4 through multiple equivalent links, fault flow information 2 is a combination of multiple fault information flows, that is, the table entries in fault information flow 2 that reach the destination address of the same service node include different outbound interface identifiers. This is shown in Table 4 below.

[0112] Table 3 Fault forwarding table

[0113] Destination address / mask Outbound interface 1.1.1.1 / 24(GPU3) GE2 / 0 / 1 1.1.1.1 / 24(GPU3) GE3 / 0 / 1 1.1.1.2 / 24(GPU4) GE2 / 0 / 1 1.1.1.2 / 24(GPU4) GE3 / 0 / 1

[0114] After receiving FLSN message 3, the controller obtains fault flow information 2 from it. According to the stored network topology and fault flow information 2, the controller determines that link 1 and link 2 are faulty. At the same time, the controller determines that GPU3 and GPU4 are unreachable.

[0115] After the controller determines that GPU3 and GPU4 are unreachable, it generates and sends a notification message 2 to the computing manager, which is used to inform the computing manager that GPU3 and GPU4 connected to device D are unreachable. The computing manager performs fault isolation or checkpoint-related operations based on notification message 2. The computing manager also generates and sends FLSN message 4 to other service nodes (GPU1 and GPU2) to inform other service nodes that GPU3 and GPU4 connected to device D are unreachable to avoid black hole transmission.

[0116] It can be understood that the FLSN message 4 may also include the fault flow information 2 .

[0117] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device corresponding to the fault handling method. Figure 3 , Figure 3 A fault handling device provided in an embodiment of the present application, the device is applied to a first network device, and the device includes a detection unit 310 and a sending unit 320;

[0118] The sending unit 320 is configured to send a first fast link state notification FLSN message to a directly connected third network device if the detection unit 310 detects a first link failure directly connected to the second network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the forwarding path after switching does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller;

[0119] The sending unit 320 is also used to send a third FLSN message to the controller if the detection unit 310 detects that the first service node accessed is unreachable, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0120] Optionally, the device further comprises:

[0121] a receiving unit (not shown in the figure), configured to receive a fourth FLSN message sent by a directly connected fourth network device, wherein the fourth FLSN message includes third fault flow information;

[0122] a determining unit (not shown in the figure), configured to determine a third link failure according to the third fault flow information, wherein the third link is a direct link between the fourth network device and the fifth network device;

[0123] a switching unit (not shown in the figure), configured to switch the forwarding path according to the third fault flow information if the receiving unit does not receive the FLSN message sent by other directly connected network devices except the fourth network device, so that the switched forwarding path passes through the other network devices but not the fourth network device;

[0124] The sending unit 320 is also used to send a fifth FLSN message to the controller if the receiving unit receives the FLSN message sent by the other network device, and the fifth FLSN message includes fourth fault flow information, so that the controller determines that the service node connected to the fourth network device is unreachable according to the fourth fault flow information, and sends a second notification message to the computing manager.

[0125] Optionally, the fault flow information includes the number of service nodes reachable through the faulty link, and the IP address and mask of each service node.

[0126] Optionally, the device further comprises:

[0127] A generating unit (not shown in the figure), configured to, when receiving a plurality of fault flow information, sequentially generate a fault forwarding table corresponding to each fault flow information, each fault forwarding table including a destination address / mask field and an outgoing interface field;

[0128] An acquisition unit (not shown in the figure), used to acquire matching routing table items and forwarding table items from a local routing table and a local forwarding table according to the destination address / mask field and the outgoing interface field;

[0129] A setting unit (not shown in the figure), used to set the routing table entry and the forwarding table entry to an invalid state;

[0130] The destination address / mask field carries the destination address and mask of the service node; the outbound interface field carries the interface identifier of the interface receiving the fault flow information, or the outbound interface field carries the interface identifier of the interface of the faulty link.

[0131] Optionally, the receiving unit (not shown in the figure) is further used to receive path information sent by the sixth network device, where the path information includes a destination address of the service node;

[0132] The device also includes: a deletion unit (not shown in the figure), which is used to delete the fault forwarding table entry if there is a fault forwarding table entry matching the destination address in the local fault forwarding table, and the fault forwarding table entry includes the interface identifier of the interface that receives the path information, and delete the routing table entry and forwarding table entry that match the destination address.

[0133] Therefore, by applying the fault handling device provided in the present application, if a first link failure directly connected to a second network device is detected, the first network device sends a first fast link state notification FLSN message to a directly connected third network device, and the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller; if it is detected that the first service node accessed is unreachable, the first network device sends a third FLSN message to the controller, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

[0134] In this way, the control plane and forwarding plane in the network device are managed separately. The control plane continues to use the original routing protocol, and the forwarding plane uses FLSN messages to implement link fault notifications to quickly perceive link changes. In this way, traffic switching and notification are achieved without the business being aware of it, greatly reducing the impact of link failures on the business. At the same time, the network device also quickly notifies the controller of the service node unreachable information, and the controller and the computing controller work together to quickly handle the problem of service node unavailability caused by network failures, which also greatly reduces the impact of network failures on large model training and reasoning processes.

[0135] Based on the same inventive concept, the embodiment of the present application also provides a network device, such as Figure 4 As shown, it includes a processor 410, a transceiver 420 and a machine-readable storage medium 430, the machine-readable storage medium 430 stores machine-executable instructions that can be executed by the processor 410, and the processor 410 is prompted by the machine-executable instructions to execute the fault handling method provided in the embodiment of the present application. Figure 3 The fault handling device shown can be used as follows Figure 4 The network device hardware structure shown is implemented.

[0136] The computer-readable storage medium 430 may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the computer-readable storage medium 430 may also be at least one storage device located away from the processor 410.

[0137] The processor 410 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.

[0138] In the embodiment of the present application, the processor 410 reads the machine executable instructions stored in the machine readable storage medium 430, and the machine executable instructions enable the processor 410 itself and the transceiver 420 to execute the fault handling method described in the aforementioned embodiment of the present application.

[0139] In addition, an embodiment of the present application provides a machine-readable storage medium 430, which stores machine-executable instructions. When called and executed by the processor 410, the machine-executable instructions prompt the processor 410 itself and the calling transceiver 420 to execute the fault handling method described in the aforementioned embodiment of the present application.

[0140] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0141] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0142] As for the fault handling device and the machine-readable storage medium embodiments, since the method contents involved are basically similar to those of the aforementioned method embodiments, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiments.

[0143] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A fault handling method, characterized in that: The method is applied to a first network device, and the method includes: If a first link failure directly connected to the second network device is detected, a first fast link state notification FLSN message is sent to a directly connected third network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when no FLSN message sent by other directly connected network devices other than the first network device is received, and the switched forwarding path does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, a second FLSN message is sent to the controller; If it is detected that the accessed service node is unreachable, a third FLSN message is sent to the controller, the third FLSN message including second fault flow information, so that the controller determines that the service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

2. The method according to claim 1, characterized in that The method further comprises: receiving a fourth FLSN message sent by a directly connected fourth network device, wherein the fourth FLSN message includes third fault flow information; Determine a third link failure according to the third fault flow information, where the third link is a direct link between the fourth network device and the fifth network device; If no FLSN message is received from directly connected network devices other than the fourth network device, switching the forwarding path according to the third fault flow information so that the switched forwarding path passes through the other network devices but not the fourth network device; If a fifth FLSN message sent by the other network device is received, a fifth FLSN message is sent to the controller, wherein the fifth FLSN message includes fourth fault flow information, so that the controller determines that the service node connected to the fifth network device is unreachable based on the fourth fault flow information, and sends a second notification message to the computing manager.

3. The method according to any one of claims 1 or 2, characterized in that: The fault flow information includes the number of service nodes reachable through the faulty link, and the IP address and mask of each service node.

4. The method according to claim 3, characterized in that The method further comprises: When multiple fault flow information is received, a fault forwarding table corresponding to each fault flow information is generated in sequence, each fault forwarding table including a destination address / mask field and an outbound interface field; According to the destination address / mask field and the outgoing interface field, obtaining matching routing table entries and forwarding table entries from a local routing table and a local forwarding table; Setting the routing table entry and the forwarding table entry to an invalid state; The destination address / mask field carries the destination address and mask of the service node; the outbound interface field carries the interface identifier of the interface receiving the fault flow information, or the outbound interface field carries the interface identifier of the interface of the faulty link.

5. The method according to claim 4, characterized in that The method further comprises: receiving path information sent by a sixth network device, wherein the path information includes a destination address of a service node; If a fault forwarding entry matching the destination address exists in the local fault forwarding table, and the fault forwarding entry includes the interface identifier of the interface receiving the path information, the fault forwarding entry is deleted, and the routing table entry and forwarding table entry matching the destination address are deleted.

6. A fault handling device, characterized in that: The device is applied to a first network device, and comprises a detection unit and a sending unit; The sending unit is used for sending a first fast link state notification FLSN message to a directly connected third network device if the detection unit detects a first link failure directly connected to the second network device, wherein the first FLSN message includes first fault flow information, so that the third network device determines the first link failure according to the first fault flow information, and switches the forwarding path according to the first fault flow information when the FLSN message sent by other directly connected network devices other than the first network device is not received, and the forwarding path after switching does not pass through the first network device; or, when receiving the FLSN message sent by other network devices, sends a second FLSN message to the controller; The sending unit is also used to send a third FLSN message to the controller if the detection unit detects that the first service node accessed is unreachable, and the third FLSN message includes second fault flow information, so that the controller determines that the first service node is unreachable according to the second fault flow information, and sends a first notification message to the computing manager.

7. The device according to claim 6, characterized in that The device also includes: A receiving unit, configured to receive a fourth FLSN message sent by a directly connected fourth network device, wherein the fourth FLSN message includes third fault flow information; a determining unit, configured to determine a third link failure according to the third fault flow information, wherein the third link is a direct link between the fourth network device and the fifth network device; a switching unit, configured to switch a forwarding path according to the third fault flow information if the receiving unit does not receive the FLSN message sent by other directly connected network devices except the fourth network device, so that the switched forwarding path passes through the other network devices but not the fourth network device; The sending unit is also used to send a fifth FLSN message to the controller if the receiving unit receives a fifth FLSN message sent by the other network devices, and the fifth FLSN message includes fourth fault flow information, so that the controller determines that the service node connected to the fifth network device is unreachable according to the fourth fault flow information, and sends a second notification message to the computing manager.

8. The device according to any one of claims 6 or 7, characterized in that The fault flow information includes the number of service nodes reachable through the faulty link, and the IP address and mask of each service node.

9. The device according to claim 8, characterized in that The device also includes: A generating unit, configured to, when receiving a plurality of fault flow information, sequentially generate a fault forwarding table corresponding to each fault flow information, each fault forwarding table including a destination address / mask field and an outgoing interface field; An acquisition unit, configured to acquire matching routing table items and forwarding table items from a local routing table and a local forwarding table according to the destination address / mask field and the outgoing interface field; A setting unit, used to set both the routing table entry and the forwarding table entry to an invalid state; The destination address / mask field carries the destination address and mask of the service node; the outbound interface field carries the interface identifier of the interface receiving the fault flow information, or the outbound interface field carries the interface identifier of the interface of the faulty link.

10. The device according to claim 9, characterized in that The receiving unit is further used to receive path information sent by the sixth network device, where the path information includes a destination address of the service node; The device also includes: a deleting unit, which is used to delete the fault forwarding entry if there is a fault forwarding entry matching the destination address in the local fault forwarding table, and the fault forwarding entry includes the interface identifier of the interface that receives the path information, and delete the routing table entry and forwarding table entry matching the destination address.

Citation Information

Patent Citations

  • Failure notification in a network having serially connected nodes

    CN101682552A

  • Link detection method and device, network equipment and controller

    CN106936655A

  • Distributing information in communication network

    CN115769549A

  • Fault processing method, related equipment and system

    CN117176546A

  • Failure notification in a network having serially connected nodes

    WO2008130620A1