Silent fault processing method and device and related equipment

Abstract

The invention relates to the technical field of intelligent calculation centers, in particular to a silent fault processing method and device and related equipment. The method comprises the steps of constructing a first matching item used for matching forward flow of a training task and a second matching item used for matching reverse flow of the training task based on source end and destination end information of the training task; and issuing the first matching item and the second matching item to each network device of a forwarding path used for forwarding the traffic of the training task, establishing a corresponding forward flow table after each network device receives the forward traffic of the training task, establishing a corresponding reverse flow table after the network device receives the reverse traffic of the training task, and forwarding the forward flow table and the reverse flow table. The forward flow table and the reverse flow table comprise aging time; receiving a forward flow table and a reverse flow table corresponding to the training task reported by each network device; and according to the forward flow table and reverse flow table information corresponding to the training task reported by each network device, detecting whether each network device has a silent fault.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent computing centers, and in particular to a silent fault handling method, apparatus, and related equipment. Background Art

[0002] With the release of various large-scale AI (Artificial Intelligence) models, demand for intelligent computing centers has exploded, ushering in a new wave of AI technology development. Mainstream large-scale models are entering an era with hundreds of billions or even trillions of parameters.

[0003] A trillion-parameter model requires a massive computing cluster with over 10,000 cards and a massive network interconnecting hundreds of network devices and nodes. Any network failure will impact the effective training time of the cluster, prolonging AI training tasks and incurring significant time costs. Silent network failures are undoubtedly among the most difficult to detect, locate, and resolve. These include device forwarding table entry anomalies, device forwarding component anomalies, routing black holes, and configuration errors. Although the device port is physically up, it cannot forward traffic. These silent network failures can cause session-level anomalies and significantly reduce AI training efficiency. Summary of the Invention

[0004] The present application provides a silent fault handling method, apparatus, and related equipment.

[0005] In a first aspect, the present application provides a silent fault handling method, applied to a controller, the method comprising:

[0006] Based on the source and destination information of the training task, construct a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task;

[0007] Sending the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, wherein the forward flow table and the reverse flow table include aging time;

[0008] Receive forward flow table and reverse flow table information corresponding to the training task reported by each network device;

[0009] Based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device, detect whether each network device has a silent failure.

[0010] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end;

[0011] The source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0012] Optionally, the forwarding path includes a forward traffic forwarding path and a reverse traffic forwarding path, and the network devices included in the forward traffic forwarding path and the reverse traffic forwarding path are the same; and the step of detecting whether a silent fault occurs according to the forward flow table and the reverse flow table fed back by each network device includes:

[0013] If the first N network devices included in the forward traffic forwarding path report forward flow tables, and the network devices other than the first N network devices do not report forward flow tables, and the network devices included in the reverse traffic forwarding path do not report reverse flow tables, it is determined that the Nth network device has a silent fault, where N is a positive integer greater than or equal to 1;

[0014] If the first M network devices included in the reverse traffic forwarding path report a reverse flow table, the other network devices except the first M network devices do not report a reverse flow table, and all network devices included in the forward traffic forwarding path report a forward flow table, then it is determined that the Mth network device has a silent failure, where M is a positive integer greater than or equal to 1.

[0015] Optionally, after determining the target network device where a silent failure occurs, the method further includes:

[0016] If it is determined that there is at least one other reachable path between the target network device and the destination access device, after eliminating the port corresponding to the current forwarding path, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0017] If it is determined that there is no other reachable path between the target network device and the destination access device, after removing the port corresponding to the current forwarding path on the previous-hop network device of the target network device, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0018] Among them, if the target network device included in the forward traffic forwarding path has a silent failure, the destination access device is the access device accessed by the destination end; if the target network device included in the reverse traffic forwarding path has a silent failure, the destination access device is the access device accessed by the source end.

[0019] In a second aspect, the present application provides a silent fault handling method, applied to a network device, the method comprising:

[0020] Receive a first matching item and a second matching item sent by a controller, wherein the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network;

[0021] After receiving the forward flow of the training task, a corresponding forward flow table is established, and after receiving the reverse flow of the training task, a corresponding reverse flow table is established, wherein the forward flow table and the reverse flow table include an aging time;

[0022] The forward flow table and reverse flow table information corresponding to the training task are reported to the controller based on a preset period, so that the controller can detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0023] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; the source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0024] Optionally, after receiving the forward flow of the training task, the step of establishing a corresponding forward flow table includes:

[0025] If the source IP address, destination IP address, and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the training task is received, and whether a forward flow table corresponding to the forward traffic exists. If it is determined that there is, the aging time of the forward flow table is refreshed; otherwise, a forward flow table corresponding to the forward traffic is established;

[0026] After receiving the reverse flow of the training task, the steps of establishing the corresponding reverse flow table include:

[0027] If the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received, and it is judged whether there is a reverse flow table corresponding to the reverse traffic. If it is determined that there is, the aging time of the reverse flow table is refreshed; otherwise, a reverse flow table corresponding to the reverse traffic is established.

[0028] Optionally, the forward flow table and the reverse flow table further include a hit mark item; the aging rules of any forward flow table and the reverse flow table include:

[0029] When the aging period of a flow table arrives, if the flow table is not hit within the aging period and the hit mark item of the flow table is hit, the hit mark item of the flow table is set to miss; if the flow table is not hit within the aging period and the hit mark item of the flow table is miss, the flow table is deleted.

[0030] In a third aspect, the present application provides a silent fault handling device, applied to a controller, the device comprising:

[0031] A construction unit, configured to construct, based on source and destination information of the training task, a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task;

[0032] A sending unit, configured to send the first matching item and the second matching item to each network device included in a forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, wherein the forward flow table and the reverse flow table include an aging time;

[0033] A receiving unit, configured to receive forward flow table and reverse flow table information corresponding to the training task reported by each network device;

[0034] The processing unit is used to detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0035] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end;

[0036] The source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0037] Optionally, the forwarding path includes a forward traffic forwarding path and a reverse traffic forwarding path, and the network devices included in the forward traffic forwarding path and the reverse traffic forwarding path are the same; when detecting whether a silent fault occurs based on the forward flow table and the reverse flow table fed back by each network device, the processing unit is specifically configured to:

[0038] If the first N network devices included in the forward traffic forwarding path report forward flow tables, and the network devices other than the first N network devices do not report forward flow tables, and the network devices included in the reverse traffic forwarding path do not report reverse flow tables, it is determined that the Nth network device has a silent fault, where N is a positive integer greater than or equal to 1;

[0039] If the first M network devices included in the reverse traffic forwarding path report a reverse flow table, the other network devices except the first M network devices do not report a reverse flow table, and all network devices included in the forward traffic forwarding path report a forward flow table, then it is determined that the Mth network device has a silent failure, where M is a positive integer greater than or equal to 1.

[0040] Optionally, the processing unit is further configured to:

[0041] If it is determined that there is at least one other reachable path between the target network device and the destination access device, after eliminating the port corresponding to the current forwarding path, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0042] If it is determined that there is no other reachable path between the target network device and the destination access device, after removing the port corresponding to the current forwarding path on the previous-hop network device of the target network device, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0043] Among them, if the target network device included in the forward traffic forwarding path has a silent failure, the destination access device is the access device accessed by the destination end; if the target network device included in the reverse traffic forwarding path has a silent failure, the destination access device is the access device accessed by the source end.

[0044] In a fourth aspect, the present application provides a silent fault handling device, applied to a network device, the device comprising:

[0045] A receiving unit, configured to receive a first matching item and a second matching item sent by a controller, wherein the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in a forwarding path for forwarding the training task traffic in the network;

[0046] an establishing unit, configured to establish a corresponding forward flow table after receiving the forward flow of the training task, and to establish a corresponding reverse flow table after receiving the reverse flow of the training task, wherein the forward flow table and the reverse flow table include an aging time;

[0047] A reporting unit is used to report the forward flow table and reverse flow table information corresponding to the training task to the controller based on a preset period, so that the controller can detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0048] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; the source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0049] Optionally, after receiving the forward flow of the training task, when establishing the corresponding forward flow table, the establishing unit is specifically configured to:

[0050] If the source IP address, destination IP address, and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the training task is received, and whether a forward flow table corresponding to the forward traffic exists. If it is determined that there is, the aging time of the forward flow table is refreshed; otherwise, a forward flow table corresponding to the forward traffic is established;

[0051] After receiving the reverse flow of the training task, when establishing the corresponding reverse flow table, the establishing unit is specifically used to:

[0052] If the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received, and it is judged whether there is a reverse flow table corresponding to the reverse traffic. If it is determined that there is, the aging time of the reverse flow table is refreshed; otherwise, a reverse flow table corresponding to the reverse traffic is established.

[0053] Optionally, the forward flow table and the reverse flow table further include a hit mark item; the aging rules of any forward flow table and the reverse flow table include:

[0054] When the aging period of a flow table arrives, if the flow table is not hit within the aging period and the hit mark item of the flow table is hit, the hit mark item of the flow table is set to miss; if the flow table is not hit within the aging period and the hit mark item of the flow table is miss, the flow table is deleted.

[0055] In a fifth aspect, an embodiment of the present application provides a silent fault handling device, the silent fault handling device comprising:

[0056] a memory for storing program instructions;

[0057] The processor is configured to call the program instructions stored in the memory and execute the steps of the method as described in any one of the first aspects above according to the obtained program instructions.

[0058] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the steps of the method described in any one of the above-mentioned first aspects.

[0059] In a seventh aspect, an embodiment of the present application provides a silent fault handling device, the silent fault handling device comprising:

[0060] a memory for storing program instructions;

[0061] The processor is used to call the program instructions stored in the memory and execute the steps of the method as described in any one of the second aspects according to the obtained program instructions.

[0062] In an eighth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the steps of the method described in any one of the above-mentioned second aspects.

[0063] In summary, the silent fault handling method provided in the embodiment of the present application is applied to a controller, and the method includes: based on the source and destination information of the training task, constructing a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task; sending the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, and the forward flow table and the reverse flow table include an aging time; receiving the forward flow table and reverse flow table information corresponding to the training task reported by each network device; and detecting whether a silent fault occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0064] The silent fault handling method provided in this embodiment automatically analyzes and determines silent faults based on the flow table's persistence status and the controller's global perspective, significantly improving problem location efficiency. This solves the problem of passively detecting silent network faults. Flow table technology enables proactive detection of silent faults, providing foresight and preventing them before they occur.

[0065] Furthermore, the issue of silent network failures requiring manual intervention for service recovery is resolved. Based on the controller's control of the hash algorithm and end-to-end forwarding paths across all network devices, the controller automatically switches forwarding paths by adjusting the UDP source port number of RoCEv2 traffic affected by silent failures. This enables automatic recovery of AI training services and ensures the long-term stable operation of AI training. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0067] Figure 1 A detailed flowchart of a silent fault handling method provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of flow table maintenance for each network device provided in an embodiment of the present application;

[0069] Figure 3 Another schematic diagram of flow table maintenance for each network device provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a fault port switching provided in an embodiment of the present application;

[0071] Figure 5 Another schematic diagram of fault port switching provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of path switching provided in an embodiment of the present application;

[0073] Figure 7 Another path switching schematic diagram provided in an embodiment of the present application;

[0074] Figure 8 A detailed flowchart of another silent fault handling method provided in an embodiment of the present application;

[0075] Figure 9A schematic diagram of the structure of a silent fault handling device provided in an embodiment of the present application;

[0076] Figure 10 A schematic diagram of the structure of another silent fault handling device provided in an embodiment of the present application;

[0077] Figure 11 A schematic diagram of the hardware architecture of a silent fault handling device provided in an embodiment of the present application;

[0078] Figure 12 A schematic diagram of the hardware architecture of another silent fault handling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0080] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may also be interpreted as "at the time of" or "when" or "in response to determining".

[0081] Currently, silent network failures are usually analyzed and troubleshooted passively based on service reports:

[0082] 1. Service fault reporting: The AI training task platform discovered sessions with long-tail latency, severely impacting training progress. The platform reported the abnormal session information to the network operations department. For example, in training task 1, the RoCE flow from GPU 1 (IP address: IP 1) to GPU 5 (IP address: IP 5) experienced multiple retransmissions for an extended period of time, even causing the training job to be interrupted (GPU 1 -> Leaf 1 -> Spine 1 -> Leaf 3 to GPU 5).

[0083] 2. Feature Traffic Collection and Analysis: Using feature flow collection technology, traffic from network devices along the entire forwarding path is collected and sent to the analyzer for analysis. For example, traffic with source IP address IP 1 and destination IP address IP 5 is collected from Leaf 1 connected to GPU 1, Leaf 3 connected to GPU 3, and all spine nodes (Spine 1 and Spine 2) and sent to the analyzer for analysis.

[0084] 3. Analyzer analysis revealed that traffic was not being forwarded after reaching Spine 1, thus narrowing the problem to Spine 1. Current analyzer systems cannot detect and analyze silent faults (such as abnormal device forwarding entries, abnormal forwarding components, routing black holes, and configuration errors). Network administrators must manually intervene to further analyze the fault, identify the root cause, and resolve the issue.

[0085] In other words, for silent network failures, after initially narrowing down the scope by collecting traffic and analyzing anomalies with an analyzer, manual intervention is required to investigate, locate the root cause, and repair the fault one by one. Troubleshooting can take from tens of minutes to several hours, severely impacting AI training services.

[0086] The embodiment of the present application proposes a solution for automatic perception of silent faults and automatic switching of affected traffic in collaboration between the end and the network (data forwarding plane and control plane). This solution actively perceives network silent faults based on the flow table technology of the data forwarding plane. Based on the control plane's global topology perspective to control the network topology and link status, combined with the analysis of the flow table's survival status, automatic analysis of silent faults and judgment of abnormal points are achieved. Based on the control plane's control of the HASH algorithm and end-to-end forwarding path of the entire network equipment, combined with the silent fault abnormal points, a forwarding path switching solution for RoCEv2 traffic affected by silent faults is automatically recommended. The SDN controller of the control plane is linked to the Agent, and the UDP source port number of the RoCEv2 traffic affected by the silent fault is automatically sent to the corresponding network card through UCCL to achieve automatic switching of the forwarding path. Greatly improve the reliability of the intelligent computing center scenario network and the efficiency of the execution of AI large model training tasks.

[0087] For example, see Figure 1 FIG. 1 is a detailed flow chart of a silent fault handling method provided in an embodiment of the present application. The method is applied to a controller and includes the following steps:

[0088] Step 100: Based on the source and destination information of the training task, construct a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task.

[0089] In embodiments of the present application, a controller (e.g., an SDN (Software Defined Network) controller) collects data such as network topology, network-wide link status, and hash algorithms for all network devices based on telemetry (a technique for remotely collecting data from physical or virtual devices at high speed). The LLDP (Link Layer Discovery Protocol) maintains the interconnection information between the GPU (Graphics Processing Unit) server and the connected Leaf switch.

[0090] The AI computing power scheduling platform creates an AI training task, selects a suitable GPU server to carry the AI training task based on information such as the number of GPU cards required, and notifies the SDN controller of the IP address and other information of the GPU server where the AI training task is located.

[0091] The collective communication library on each GPU server initializes the communication domain. Based on the communication algorithm, GPU cards establish socket links, exchange key information such as memory addresses through sockets, and establish the QP (Queue Pair) required for subsequent RoCE (RDMA over Converged Ethernet, a network protocol that allows applications to implement remote memory access over Ethernet) communication.

[0092] The agent on each GPU server reads the communication relationship between GPU cards of the AI training task and reports it to the SDN controller. The reported information includes: task ID, communication algorithm (RING, Tree, HD, etc.) and communication domain list (communication domain ID, network card IP address for communication between GPU cards, source QP number and destination QP number).

[0093] In an embodiment of the present application, after receiving the inter-GPU card communication relationship of the AI training task included on the GPU server, the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information included in the inter-GPU card communication relationship of the AI training task.

[0094] In an embodiment of the present application, a preferred implementation method is: the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; the source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0095] That is to say, taking the source end as GPU card 1 of GPU server 1 (network card IP address is IP 1), the destination end as GPU card 5 of GPU server 5 (network card IP address is IP 5), the source end QP is QP 1, and the destination end QP is QP 5 as an example for explanation, for example, refer to Table 1, which is an example of ACL matching items (first matching items) for forward traffic; refer to Table 2, which is an example of ACL matching items (second matching items) for reverse traffic.

[0096] Matches UDP port SrcIP DestIP Dest QP Value 4791 IP 1 IP 5 QP 5

[0097] Table 1

[0098] Matches UDP port SrcIP DestIP Dest QP Value 4791 IP 5 IP 1 QP 1

[0099] Table 2

[0100] After receiving the inter-GPU communication relationships for AI training tasks reported by the GPU server, the SDN controller selects a forwarding path for the training task based on collected data such as the network topology, network-wide link status, and the hash algorithm of all network devices, combined with the maintained interconnection information between the GPU server and the access leaf switch. Subsequent training task traffic is forwarded using the network devices included in this forwarding path.

[0101] Step 110: Send the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network.

[0102] Among them, after receiving the forward traffic of the training task, each network device establishes a corresponding forward flow table, and after receiving the reverse traffic of the training task, it establishes a corresponding reverse flow table. The forward flow table and the reverse flow table include aging time.

[0103] After determining the network devices included in the network for forwarding AI training tasks, the SDN controller sends the first matching item and the second matching item to each network device. After receiving the AI training task, if the source IP address, destination IP address and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the AI training task is received. Then, it is determined whether there is a forward flow table corresponding to the forward traffic. If not, the traffic message needs to be copied and sent to the CPU of the network device for processing to establish the forward flow table corresponding to the AI training task; if it exists, the aging time of the forward flow table is refreshed.

[0104] Similarly, if the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received. Then, it is determined whether there is a reverse flow table corresponding to the reverse traffic. If not, the traffic message needs to be copied and sent to the CPU of this network device for processing to establish the reverse flow table corresponding to the AI training task; if it exists, the aging time of the reverse flow table is refreshed.

[0105] In an embodiment of the present application, the forward flow table and the reverse flow table also include a hit mark item; then, in actual application, a preferred aging rule for the forward flow table and the reverse flow table includes: when the aging cycle of a flow table arrives, if the flow table is not hit within the aging cycle, and the hit mark item of the flow table is a hit, then the hit mark item of the flow table is set to a miss; if the flow table is not hit within the aging cycle, and the hit mark item of the flow table is a miss, then the flow table is deleted.

[0106] That is to say, if a network device receives a traffic packet, and the flow table corresponding to the traffic packet does not exist locally on the network device, a corresponding flow table needs to be established. If a flow table corresponding to the traffic packet exists, the aging time of the flow table is refreshed, and the hit mark item is in the hit state; if the traffic packet corresponding to the flow table is not received for two consecutive aging cycles, the flow table is deleted.

[0107] Still taking the source end as GPU card 1 of GPU server 1 (network card IP address is IP 1), the destination end as GPU card 5 of GPU server 5 (network card IP address is IP 5), the source end QP is QP 1, and the destination end QP is QP 5 as an example for explanation, for example, see Table 3, which is an example of the forward flow table format; see Table 3.

[0108] Table 4 shows an example of the reverse flow table format.

[0109]

[0110] Table 3

[0111]

[0112] Table 4

[0113] Dest IP refers to the destination IP address; Src IP refers to the source IP address; Dest QP refers to the destination QP; Aging refers to the aging time; and Hit refers to the hit status.

[0114] Step 120: Receive the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0115] In the embodiment of the present application, when the flow table aging period (T) expires, each network device reports the flow table information maintained by each network device to the SDN controller through a preset protocol.

[0116] Step 130: Detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0117] Specifically, the SDN controller determines whether a silent failure occurs on each network device based on the forward flow table and reverse flow table information corresponding to the AI training tasks reported by each network device.

[0118] In an embodiment of the present application, the forwarding path includes a forward traffic forwarding path and a reverse traffic forwarding path, and the network devices included in the forward traffic forwarding path and the reverse traffic forwarding path are the same.

[0119] Then, in the embodiment of the present application, when detecting whether a silent failure occurs based on the forward flow table and the reverse flow table fed back by each network device, a preferred implementation method is:

[0120] If the first N network devices included in the forward traffic forwarding path report forward flow tables, and the network devices other than the first N network devices do not report forward flow tables, and the network devices included in the reverse traffic forwarding path do not report reverse flow tables, then it is determined that a silent failure has occurred in the Nth network device, where N is a positive integer greater than or equal to 1.

[0121] For example, consider the intercommunication between GPU 1 and GPU 5. Packets sent by GPU 1 (forward traffic) are forwarded to GPU 5 via Leaf 1, Spine 1, and Leaf 3. Packets sent by GPU 5 (reverse traffic) are forwarded to GPU 1 via Leaf 3, Spine 1, and Leaf 1.

[0122] Silent fault mode 1: The forward flow tables reported by each network device are only reported by the first N devices (N ≥ 1) in the forward traffic forwarding path (Leaf 1-Spine 1-Leaf 3). No network device reports a reverse flow table. This indicates a silent outbound forwarding failure on the last device with a forward flow table in the forward traffic forwarding path.

[0123] For example, see Figure 2As shown in the figure, Leaf 1 in the forward traffic forwarding path reports the forward flow table corresponding to the training task, while Spine 1 and Leaf 3 do not report the forward flow table corresponding to the training task, and Leaf 3, Spine 1, and Leaf 1 in the reverse traffic forwarding path do not report the reverse flow table corresponding to the training task. It is determined that a silent fault has occurred on Leaf 1, that is, Leaf 1 cannot forward the flow table of the training task to Spine 1.

[0124] If the first M network devices included in the reverse traffic forwarding path report a reverse flow table, the other network devices except the first M network devices do not report a reverse flow table, and all network devices included in the forward traffic forwarding path report a forward flow table, then it is determined that the Mth network device has a silent failure, where M is a positive integer greater than or equal to 1.

[0125] Silent failure mode 2: All network devices in the forward traffic forwarding path (Leaf 1-Spine 1-Leaf 3) report forward flow tables, but only the first M (M ≥ 1) network devices in the reverse traffic forwarding path (Leaf 3-Spine 1-Leaf 1) report reverse flow tables. This indicates that a silent outbound forwarding failure has occurred on the last device in the reverse traffic forwarding path that reports a reverse flow table.

[0126] For example, see Figure 3 As shown in the figure, Leaf 1, Spine 1, and Leaf 3 in the forward traffic forwarding path all report the forward flow table corresponding to the training task. Leaf 3 and Spine 1 in the reverse traffic forwarding path report the reverse flow table corresponding to the training task. Leaf 1 does not report the reverse flow table corresponding to the training task. It is determined that a silent fault has occurred on Spine 1, that is, Spine 1 cannot forward the reverse traffic of the training task to Leaf 1.

[0127] In the embodiment of the present application, after determining the target network device where a silent fault occurs, the silent fault handling method may further include the following steps:

[0128] Case 1: If it is determined that there is at least one other reachable path between the target network device and the destination access device, after eliminating the port corresponding to the current forwarding path, a port with the lightest load is selected from the remaining multiple ports that can reach the destination access device as the port for traffic switching.

[0129] It should be noted that if the target network device included in the forward traffic forwarding path has a silent failure, the destination access device is the access device accessed by the destination end; if the target network device included in the reverse traffic forwarding path has a silent failure, the destination access device is the access device accessed by the source end.

[0130] Specifically, let's take the intercommunication between GPU 1 and GPU 5 as an example. The packets sent by GPU 1 (forward traffic) are forwarded to the destination GPU 5 via Leaf 1, Spine 1, and Leaf 3. The packets sent by GPU 5 (reverse traffic) are forwarded to the destination GPU 1 via Leaf 3, Spine 1, and Leaf 1.

[0131] If the target network device (e.g., Leaf 1) included in the forward traffic forwarding path experiences a silent failure, the destination access device is Leaf 3 connected to GPU 5; if the target network device (e.g., Spine 1) included in the reverse traffic forwarding path experiences a silent failure, the destination access device is Leaf 1 connected to GPU 1.

[0132] For example, see Figure 4 As shown in the figure, assuming that there are multiple paths from the target network device with a silent failure (e.g., Leaf 1) to the destination Leaf device (e.g., Leaf 3), then it is only necessary to switch the port of the target network device with the silent failure that is used to forward the training task traffic. Specifically, after the SDN controller removes the port that currently has a silent failure (e.g., interface 1 of Leaf 1), it selects the port with the lightest load from the remaining multiple ports that can reach the destination Leaf (e.g., Leaf 3) as the port for traffic switching (e.g., interface 2 of Leaf 1). In this way, the training task traffic is not forwarded from the port where the silent failure occurs.

[0133] Case 2: If it is determined that there is no other reachable path between the target network device and the destination access device, after removing the port corresponding to the current forwarding path on the previous-hop network device of the target network device, a port with the lightest load is selected from the remaining multiple ports that can reach the destination access device as the port for traffic switching.

[0134] For example, see Figure 5As shown in the figure, assuming that there is only one path from the target network device (e.g., Spine 1) with a silent failure to the destination Leaf device (e.g., Leaf 1) (e.g., there is only one link from Spine 1 to Leaf 1), it is necessary to switch the forwarding port on the previous-hop network device (e.g., Leaf 3) of the silent failure device. Specifically, after the previous-hop network device (e.g., Leaf 3) removes the port corresponding to the current forwarding (e.g., Leaf 3's interface 1), the SDN controller selects the lightest-loaded port (e.g., Leaf 3's interface 2) from the remaining multiple ports that can reach the destination Leaf (e.g., Leaf 1) as the port to switch traffic to.

[0135] In an embodiment of the present application, after determining the new forwarding path, it is necessary to determine the UDP source port number that the training task traffic corresponding to the new forwarding path needs to carry.

[0136] Specifically, the SDN controller sends the Source Port corresponding to the new path to the Agent. The SDN controller sends the UDP source port number corresponding to the characteristic traffic to the Agent on the GPU server via the REST API. The characteristic traffic information includes the source IP address, destination IP address, Src QP, Dest QP, and so on. The Agent then configures the corresponding Source Port to the end-side NIC. The Agent calls UCCL via the REST API to assign the Source Port corresponding to the characteristic flow to the source-side NIC, thereby automatically switching the traffic path for silent failures.

[0137] For example, see Figure 6 As shown, it is a schematic diagram of a path switching effect provided by an embodiment of the present application; for a device with a silent fault, there are multiple paths to the destination Leaf device (Leaf 1 has paths corresponding to the two ports Interface 1 and Interface 2 to Leaf 3), and only the port of the device where the silent fault is located needs to be switched (traffic is switched from Interface 1 of Leaf 1 to Interface 2).

[0138] For example, see Figure 7 As shown, a schematic diagram of a path switching effect provided by an embodiment of the present application shows that for a device with a silent failure (Spine 1) reaching the destination Leaf device, there is only one path (as shown in the figure, there is only one link corresponding to port interface 1 from Spine1 to Leaf1), and it is necessary to switch the forwarding port on the previous-hop network device (Leaf 3) of the silent failure device (switch from port interface 1 of Leaf 3 to interface 2).

[0139] Furthermore, in an embodiment of the present application, the SDN controller synchronizes the RoCEv2 traffic information to the analyzer, and the analyzer visually compares and presents the end-to-end forwarding paths and UDP port numbers corresponding to the characteristic flows (source IP address, destination IP address, Dest QP, Src QP) before and after the switching.

[0140] For example, see Figure 8 FIG. 1 is a detailed flowchart of a silent fault handling method provided in an embodiment of the present application. The method is applied to a network device and includes the following steps:

[0141] Step 800: Receive a first matching item and a second matching item sent by a controller.

[0142] The controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network;

[0143] Step 810: After receiving the forward flow of the training task, a corresponding forward flow table is established, and after receiving the reverse flow of the training task, a corresponding reverse flow table is established.

[0144] Wherein, the forward flow table and the reverse flow table include aging time;

[0145] Step 820: Report the forward flow table and reverse flow table information corresponding to the training task to the controller based on a preset period.

[0146] Specifically, the controller detects whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0147] In the embodiment of the present application, after receiving the forward flow of the training task, when establishing the corresponding forward flow table, a preferred implementation method is:

[0148] If the source IP address, destination IP address, and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the training task is received, and whether a forward flow table corresponding to the forward traffic exists. If it is determined that there is, the aging time of the forward flow table is refreshed; otherwise, a forward flow table corresponding to the forward traffic is established;

[0149] After receiving the reverse flow of the training task, when establishing the corresponding reverse flow table, a better implementation method is:

[0150] If the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received, and it is judged whether there is a reverse flow table corresponding to the reverse traffic. If it is determined that there is, the aging time of the reverse flow table is refreshed; otherwise, a reverse flow table corresponding to the reverse traffic is established.

[0151] Optionally, the forward flow table and the reverse flow table further include a hit mark item; the aging rules of any forward flow table and the reverse flow table include:

[0152] When the aging period of a flow table arrives, if the flow table is not hit within the aging period and the hit mark item of the flow table is hit, the hit mark item of the flow table is set to miss; if the flow table is not hit within the aging period and the hit mark item of the flow table is miss, the flow table is deleted.

[0153] For example, see Figure 9 FIG. 1 is a schematic diagram of a silent fault handling device provided in an embodiment of the present application. The device is applied to a controller and includes:

[0154] A construction unit 90 is configured to construct, based on source and destination information of the training task, a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task;

[0155] A sending unit 91 is used to send the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, and the forward flow table and the reverse flow table include an aging time;

[0156] A receiving unit 92 is configured to receive forward flow table and reverse flow table information corresponding to the training task reported by each network device;

[0157] The processing unit 93 is used to detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0158] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end;

[0159] The source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0160] Optionally, the forwarding path includes a forward traffic forwarding path and a reverse traffic forwarding path, and the network devices included in the forward traffic forwarding path and the reverse traffic forwarding path are the same; when detecting whether a silent fault occurs based on the forward flow table and the reverse flow table fed back by each network device, the processing unit 93 is specifically configured to:

[0161] If the first N network devices included in the forward traffic forwarding path report forward flow tables, and the network devices other than the first N network devices do not report forward flow tables, and the network devices included in the reverse traffic forwarding path do not report reverse flow tables, it is determined that the Nth network device has a silent fault, where N is a positive integer greater than or equal to 1;

[0162] If the first M network devices included in the reverse traffic forwarding path report a reverse flow table, the other network devices except the first M network devices do not report a reverse flow table, and all network devices included in the forward traffic forwarding path report a forward flow table, then it is determined that the Mth network device has a silent failure, where M is a positive integer greater than or equal to 1.

[0163] Optionally, the processing unit 93 is further configured to:

[0164] If it is determined that there is at least one other reachable path between the target network device and the destination access device, after eliminating the port corresponding to the current forwarding path, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0165] If it is determined that there is no other reachable path between the target network device and the destination access device, after removing the port corresponding to the current forwarding path on the previous-hop network device of the target network device, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching;

[0166] Among them, if the target network device included in the forward traffic forwarding path has a silent failure, the destination access device is the access device accessed by the destination end; if the target network device included in the reverse traffic forwarding path has a silent failure, the destination access device is the access device accessed by the source end.

[0167] For example, see Figure 10 FIG. 1 is a schematic diagram of a silent fault handling device provided by the present application, which is applied to a network device and includes:

[0168] A receiving unit 100 is configured to receive a first matching item and a second matching item sent by a controller, wherein the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in a forwarding path for forwarding the training task traffic in the network;

[0169] An establishing unit 101 is configured to establish a corresponding forward flow table after receiving the forward flow of the training task, and to establish a corresponding reverse flow table after receiving the reverse flow of the training task, wherein the forward flow table and the reverse flow table include an aging time;

[0170] The reporting unit 102 is used to report the forward flow table and reverse flow table information corresponding to the training task to the controller based on a preset period, so that the controller can detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

[0171] Optionally, the source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; the source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

[0172] Optionally, after receiving the forward flow of the training task, when establishing the corresponding forward flow table, the establishing unit 101 is specifically configured to:

[0173] If the source IP address, destination IP address, and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the training task is received, and whether a forward flow table corresponding to the forward traffic exists. If it is determined that there is, the aging time of the forward flow table is refreshed; otherwise, a forward flow table corresponding to the forward traffic is established;

[0174] After receiving the reverse flow of the training task, when establishing the corresponding reverse flow table, the establishing unit 101 is specifically used to:

[0175] If the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received, and it is judged whether there is a reverse flow table corresponding to the reverse traffic. If it is determined that there is, the aging time of the reverse flow table is refreshed; otherwise, a reverse flow table corresponding to the reverse traffic is established.

[0176] Optionally, the forward flow table and the reverse flow table further include a hit mark item; the aging rules of any forward flow table and the reverse flow table include:

[0177] When the aging period of a flow table arrives, if the flow table is not hit within the aging period and the hit mark item of the flow table is hit, the hit mark item of the flow table is set to miss; if the flow table is not hit within the aging period and the hit mark item of the flow table is miss, the flow table is deleted.

[0178] The above units may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a unit is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0179] Furthermore, the silent fault handling device provided in the embodiment of the present application, from the hardware level, the hardware architecture diagram of the silent fault handling device can be found in Figure 11 As shown, the silent fault processing device may include: a memory 110 and a processor 111,

[0180] The memory 110 is used to store program instructions. The processor 111 calls the program instructions stored in the memory 110 and executes the above method embodiment applied to the controller according to the obtained program instructions. The specific implementation method and technical effects are similar and will not be repeated here.

[0181] Optionally, the present application also provides a controller comprising at least one processing element (or chip) for executing the above method embodiment applied to the controller.

[0182] Optionally, the present application also provides a program product, such as a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the above-mentioned method embodiment applied to the controller.

[0183] Furthermore, the silent fault handling device provided in the embodiment of the present application, from the hardware level, the hardware architecture diagram of the silent fault handling device can be found in Figure 12 As shown, the silent fault processing device may include: a memory 120 and a processor 121,

[0184] The memory 120 is used to store program instructions. The processor 121 calls the program instructions stored in the memory 120 and executes the above method embodiment applied to the network device according to the obtained program instructions. The specific implementation method and technical effect are similar and will not be repeated here.

[0185] Optionally, the present application also provides a network device, comprising at least one processing element (or chip) for executing the above-mentioned method embodiment applied to the network device.

[0186] Optionally, the present application also provides a program product, such as a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the above-mentioned method embodiment applied to the network device.

[0187] Here, the machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage media, or a combination thereof.

[0188] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0189] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0190] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0191] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0194] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A silent fault handling method, characterized in that: Applied to a controller, the method includes: Based on the source and destination information of the training task, construct a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task; Sending the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, wherein the forward flow table and the reverse flow table include aging time; Receive forward flow table and reverse flow table information corresponding to the training task reported by each network device; Based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device, detect whether each network device has a silent failure.

2. The method according to claim 1, wherein The source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; The source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

3. The method according to claim 1 or 2, wherein: The forwarding path includes a forward traffic forwarding path and a reverse traffic forwarding path, and the network devices included in the forward traffic forwarding path and the reverse traffic forwarding path are the same; The steps to detect whether a silent fault has occurred based on the forward and reverse flow tables reported by each network device include: If the first N network devices included in the forward traffic forwarding path report forward flow tables, and the network devices other than the first N network devices do not report forward flow tables, and the network devices included in the reverse traffic forwarding path do not report reverse flow tables, it is determined that the Nth network device has a silent fault, where N is a positive integer greater than or equal to 1; If the first M network devices included in the reverse traffic forwarding path report a reverse flow table, the other network devices except the first M network devices do not report a reverse flow table, and all network devices included in the forward traffic forwarding path report a forward flow table, then it is determined that the Mth network device has a silent failure, where M is a positive integer greater than or equal to 1.

4. The method according to claim 3, wherein After determining the target network device where the silent failure occurs, the method further includes: If it is determined that there is at least one other reachable path between the target network device and the destination access device, after eliminating the port corresponding to the current forwarding path, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching; If it is determined that there is no other reachable path between the target network device and the destination access device, after removing the port corresponding to the current forwarding path on the previous-hop network device of the target network device, select a port with the lightest load from the remaining multiple ports that can reach the destination access device as the port for traffic switching; Among them, if the target network device included in the forward traffic forwarding path has a silent failure, the destination access device is the access device accessed by the destination end; if the target network device included in the reverse traffic forwarding path has a silent failure, the destination access device is the access device accessed by the source end.

5. A silent fault handling method, characterized in that: Applied to a network device, the method includes: Receive a first matching item and a second matching item sent by a controller, wherein the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in the forwarding path for forwarding the training task traffic in the network; After receiving the forward flow of the training task, a corresponding forward flow table is established, and after receiving the reverse flow of the training task, a corresponding reverse flow table is established, wherein the forward flow table and the reverse flow table include an aging time; The forward flow table and reverse flow table information corresponding to the training task are reported to the controller based on a preset period, so that the controller can detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

6. The method according to claim 5, wherein The source IP address of the first matching item is the IP address of the source end, the destination IP address is the IP address of the destination end, and the destination QP is the QP of the destination end; the source IP address of the second matching item is the IP address of the destination end, the destination IP address is the IP address of the source end, and the destination QP is the QP of the source end.

7. The method according to claim 6, wherein After receiving the forward flow of the training task, the steps of establishing the corresponding forward flow table include: If the source IP address, destination IP address, and destination QP of the received traffic message match the first matching item, it is determined that the forward traffic of the training task is received, and whether a forward flow table corresponding to the forward traffic exists. If it is determined that there is, the aging time of the forward flow table is refreshed; otherwise, a forward flow table corresponding to the forward traffic is established; After receiving the reverse flow of the training task, the steps of establishing the corresponding reverse flow table include: If the source IP address, destination IP address and destination QP of the received traffic message match the second matching item, it is determined that the reverse traffic of the training task is received, and it is judged whether there is a reverse flow table corresponding to the reverse traffic. If it is determined that there is, the aging time of the reverse flow table is refreshed; otherwise, a reverse flow table corresponding to the reverse traffic is established.

8. The method according to any one of claims 5 to 7, wherein: The forward flow table and the reverse flow table also include hit mark items; the aging rules of any forward flow table and reverse flow table include: When the aging period of a flow table arrives, if the flow table is not hit within the aging period and the hit mark item of the flow table is hit, the hit mark item of the flow table is set to miss; if the flow table is not hit within the aging period and the hit mark item of the flow table is miss, the flow table is deleted.

9. A silent fault handling device, characterized in that: Applied to a controller, the device comprises: A construction unit, configured to construct, based on source and destination information of the training task, a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task; A sending unit, configured to send the first matching item and the second matching item to each network device included in a forwarding path for forwarding the training task traffic in the network, wherein each network device establishes a corresponding forward flow table after receiving the forward traffic of the training task, and establishes a corresponding reverse flow table after receiving the reverse traffic of the training task, wherein the forward flow table and the reverse flow table include an aging time; A receiving unit, configured to receive forward flow table and reverse flow table information corresponding to the training task reported by each network device; The processing unit is used to detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

10. A silent fault handling device, characterized in that: Applied to network equipment, the device includes: A receiving unit, configured to receive a first matching item and a second matching item sent by a controller, wherein the controller constructs a first matching item for matching the forward traffic of the training task and a second matching item for matching the reverse traffic of the training task based on the source and destination information of the training task, and sends the first matching item and the second matching item to each network device included in a forwarding path for forwarding the training task traffic in the network; an establishing unit, configured to establish a corresponding forward flow table after receiving the forward flow of the training task, and to establish a corresponding reverse flow table after receiving the reverse flow of the training task, wherein the forward flow table and the reverse flow table include an aging time; A reporting unit is used to report the forward flow table and reverse flow table information corresponding to the training task to the controller based on a preset period, so that the controller can detect whether a silent failure occurs in each network device based on the forward flow table and reverse flow table information corresponding to the training task reported by each network device.

11. A silent fault handling device, characterized in that: The silent fault processing device includes: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory, and execute the steps of the method according to any one of claims 1 to 4 or any one of claims 5 to 8 according to the obtained program instructions.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the steps of the method according to any one of claims 1 to 4 or any one of claims 5 to 8.

Images