Fault locating method and related device
By sending and receiving packets carrying detection marks in the bus switched network, analyzing packet loss situations, and quickly locate the fault location, the problem of low fault location efficiency in the computing cluster is solved, the efficiency and accuracy of fault location are improved, and the Compute the continuous serviceable time of the cluster.
Patent Information
- Application Number
- PCT/CN2023/135386
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-31
- Filing Date
- 2023-11-30
- Publication Date
- 2024-12-05
AI Technical Summary
As the scale of computing clusters grows, the efficiency of fault location in the bus switched network is low, resulting in a decrease in continuous serviceable time, and existing tools are difficult to meet the fault location requirements during memory access.
It provides a fault location method, which transmits and receives packets carrying detection marks in a bus switched network through a detection controller, analyzes packet loss situations, and quickly locates fault locations. It is suitable for networks with packet-by-packet load sharing method.
It improves the efficiency and accuracy of fault location, reduces resource waste, improves resource utilization, meets business needs, and ensures continuous serviceable time of the computing cluster.
Smart Images

Figure CN2023135386_05122024_PF_FP_ABST
Abstract
Description
A fault location method and related equipment
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 31, 2023, with application number 202310639272.5 and invention name “A method for locating faults in a bus data exchange system”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 31, 2023, with application number 202310961572.5 and invention name “A method for locating faults in a bus data exchange system”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a fault location method, a detection controller, a computing cluster, a computer-readable storage medium, and a computer program product. Background Art
[0003] With the rise of high-performance computing (HPC) and artificial intelligence (AI) technologies, the demand for computing power continues to grow. As chips gradually reach the limits of Moore's Law, the computing power (i.e., computing power) of a single chip is difficult to scale up vertically. Increasing computing power typically requires scaling out horizontally. For example, computing nodes, including central processing units (CPUs) and neural processing units (NPUs), can be connected via bus-switched networks to form computing clusters.
[0004] The computing nodes in a computing cluster can achieve memory mutual access between computing nodes through a bus switching network. In other words, memory access has evolved from intra-node access to cross-node access. The bus switching network includes switching nodes, and each switching node may include at least one switching chip. When the CPU of a computing node (such as computing node 1) accesses the memory data of the NPU of another computing node (such as computing node 2), the CPU of computing node 1 can send a data access command. The switching chip connected to the computing node 1 can convert the data access command into a data request message and route the data request message to the switching chip connected to the NPU of the computing node 2. The switching chip connected to the NPU of computing node 2 restores the data request message into a data access command and transmits it to the NPU of computing node 2, thereby completing a cross-node memory access.
[0005] As the scale of computing clusters (e.g., server chip and bus sizes) continues to grow, the number of switching chips and chip interfaces traversed by a single cross-node memory access increases dramatically, increasing the likelihood of bus switching network transmission failures. When a bus switching network failure occurs, quickly locating the fault to maximize cluster uptime becomes a pressing issue.
[0006] Summary of the Invention
[0007] This application provides a fault location method that, when a bus switching network fails, can quickly locate the fault based on the number of messages carrying detection tags received and sent by a switching node, thereby improving cluster continuous service time. This application also provides a detection controller, computing cluster, computer-readable storage medium, and computer program product corresponding to the above method.
[0008] In a first aspect, the present application provides a fault location method. This method can be executed by a detection controller. The detection controller can be software, which can be deployed on a computing node or switching node in a computing cluster, or on an independent node. The computing node, switching node, or independent node executes the program code of the software, thereby executing the fault location method of the present application. The detection controller can also be hardware, for example, the detection controller can be a server with fault location capabilities, which executes the fault location method of the present application when running.
[0009] Specifically, a detection controller sends a detection configuration to a first node in a computing cluster, where the detection configuration includes information about a second node that is being detected by the first node. The computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes. The first node and the second node can be any computing node in the multiple computing nodes or any switching node in the bus switching network. The detection controller then obtains the number of detection-tagged messages received and the number of detection-tagged messages sent by at least one switching node. The at least one switching node is a switching node in the bus switching network through which the first node initiates detection from the second node, and the detection-tagged messages are sent by the first node according to the detection configuration. The detection controller then determines the fault location in the bus switching network based on the number of detection-tagged messages received and the number of detection-tagged messages sent by the at least one switching node.
[0010] This method analyzes the number of probe-tagged packets received and sent by at least one switching node in a bus switching network to analyze packet loss of probe-tagged packets at the switching node, thereby locating faults in the bus switching network. Even if the bus switching network uses per-packet load balancing, resulting in uncertain, unpredictable, and non-traceable forwarding paths, the method can still locate packet loss and thus faults, meeting service requirements.
[0011] In some possible implementations, the detection controller may perform fault location based on node-level message transmission and reception. Specifically, when the number of detection-tagged messages received by a first switching node among at least one switching node is not equal to the number of detection-tagged messages sent by the first switching node, the detection controller may determine that the fault location in the bus switching network includes the first switching node.
[0012] This method determines the packet loss situation by analyzing the number of messages carrying detection tags sent and received by the same switching node. Based on the packet loss situation, it can be determined whether the switching node has a fault. In this way, the fault location of the node type fault can be determined more quickly, achieving efficient fault location.
[0013] In some possible implementations, the detection controller can also detect the upstream and downstream message reception and transmission conditions to locate the fault. Specifically, the bus switching network includes a first switching node and a second switching node, and the output interface of the first switching node is connected to the input interface of the second switching node. Based on this, the first switching node can be regarded as an upstream device of the second switching node, and the second switching node can be regarded as a downstream device of the first switching node. When the number of messages carrying the detection mark sent by the first switching node through the output interface (the number of messages sent upstream) is not equal to the number of messages carrying the detection mark received by the second switching node through the input interface, the detection controller determines that the fault location in the bus switching network includes the link from the output interface of the first switching node to the input interface of the second switching node.
[0014] The method determines the packet loss situation by analyzing the number of packets carrying detection tags sent by upstream nodes through outgoing interfaces and the number of packets carrying detection tags received by downstream nodes through incoming interfaces in switching nodes connected to interfaces. Based on the packet loss situation, it can be determined whether a link between the switching nodes has failed. In this way, the fault location of link-type faults can be determined relatively quickly, achieving efficient fault location.
[0015] In some possible implementations, the first node may be any computing node among the multiple computing nodes, and the detection configuration further includes a detection mode. When the detection mode is global detection, service packets of the first node carry a detection tag, and the packets carrying the detection tag include memory detection packets and service packets carrying the detection tag.
[0016] In this way, fault detection can be performed while the business is running without having to wait for the business to be suspended. Moreover, carrying the detection tag with the business message can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and realize resource reuse.
[0017] In some possible implementations, the detection controller may trigger fault location when a detection task detects a fault. Specifically, the detection controller receives a fault record reported by the first node or the second node and determines the fault location in the bus switching network based on the number of messages carrying the detection tag received and the number of messages carrying the detection tag sent by at least one switching node.
[0018] The method performs fault location according to the number of messages carrying detection tags received and sent by the switching node when a fault is detected, thereby avoiding resource waste caused by fault location when no fault occurs and improving resource utilization.
[0019] In some possible implementations, the probe configuration also includes an expected probe result. The fault record is reported when the first or second node detects an inconsistency between the expected probe result and the actual probe result. If the probe access mode is read, the expected probe result can be the data requested to be read, which can be pre-agreed data. If the probe access mode is write, the expected probe result can be the content of the probe request, such as the data to be written.
[0020] In this method, the first node or the second node can realize fault detection by comparing the consistency of the expected detection result and the actual detection result, and when a fault is detected, report the fault record to the detection controller so that the detection controller can trigger fault location, thereby improving the fault location efficiency.
[0021] In some possible implementations, the number of detection-tagged packets sent by a first node to a second node is greater than or equal to the number of paths between the first node and the second node. For example, the number of detection-tagged packets sent by a first node to any second node is greater than or equal to the number of load-sharing paths between the first node and the second node. This maximizes coverage of paths between the first node and the second node, improving the comprehensiveness of fault location analysis and, consequently, the accuracy of fault location.
[0022] In some possible implementations, the detection controller is independent of the computing cluster. This prevents the detection controller from occupying business resources in the computing cluster while performing fault location, thus ensuring normal business operation.
[0023] In some possible implementations, the detection controller can be a node in the computing cluster, or the detection controller can be deployed on a node in the computing cluster. This can reuse existing nodes, reduce system complexity, and reduce additional network bandwidth overhead.
[0024] In a second aspect, the present application provides a fault location method, which is applied to a computing cluster including a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes.
[0025] Specifically, a first node in a computing cluster receives a detection configuration sent by a detection controller, where the detection configuration includes information about a second node that is detected by the first node. The first node and the second node are any computing node among the multiple computing nodes or any switching node in the bus switching network. Then, the first node sends a message carrying a detection tag to the second node according to the detection configuration, and at least one switching node sends the number of messages carrying the detection tag received and sent by at least one switching node to the detection controller. The at least one switching node is a switching node in the bus switching network through which the first node initiates detection from the second node.
[0026] This method analyzes the number of probe-tagged packets received and sent by at least one switching node in a bus switching network to analyze packet loss of probe-tagged packets at the switching node, thereby locating faults in the bus switching network. Even if the bus switching network uses per-packet load balancing, resulting in uncertain, unpredictable, and non-traceable forwarding paths, the method can still locate packet loss and thus faults, meeting service requirements.
[0027] In some possible implementations, the first node includes a detection agent, and the switching node includes a statistics agent. Accordingly, the detection agent in the first node sends a message carrying a detection tag to the second node based on the detection configuration; and the statistics agent in the at least one switching node sends to the detection controller a number of detection-tagged messages received and sent by the at least one switching node.
[0028] The detection agent device, the statistical agent device and the detection controller can cooperate to achieve fault location.
[0029] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration further includes a detection mode. When the detection mode is global detection, the first node carries the detection flag in a service message. The message carrying the detection flag includes a memory detection message and a service message carrying the detection flag.
[0030] In this way, fault detection can be performed while the business is running without having to wait for the business to be suspended. Moreover, carrying the detection tag with the business message can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and realize resource reuse.
[0031] In some possible implementations, the detection configuration further includes an expected detection result. When the first node or the second node detects that the expected detection result is inconsistent with the actual detection result, a fault record is reported to the detection controller. The fault record is used to trigger the detection controller to perform fault location.
[0032] The method performs fault location when a fault is detected, thereby avoiding resource waste caused by performing fault location when no fault occurs and improving resource utilization.
[0033] In some possible implementations, the number of paths from the first node to the second node is M, where M is a positive integer. Accordingly, the first node may send N messages carrying a detection flag to the second node according to the detection configuration, where N is greater than or equal to M. In this way, the paths from the first node to the second node can be covered as much as possible, thereby improving the comprehensiveness of fault location analysis and thereby enhancing the accuracy of fault location.
[0034] In a third aspect, the present application provides a detection controller. The detection controller includes:
[0035] a communication module, configured to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster including a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node being any computing node among the plurality of computing nodes or any switching node in the bus switching network;
[0036] The communication module is further configured to obtain a number of messages carrying a detection tag received by and a number of messages carrying a detection tag sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates a detection from the second node, and the messages carrying the detection tag are sent by the first node according to the detection configuration;
[0037] A fault location module is configured to determine a fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.
[0038] In some possible implementations, the fault location module is specifically configured to:
[0039] When the number of messages carrying the detection tag received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection tag sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.
[0040] In some possible implementations, the bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically configured to:
[0041] When the number of packets carrying the detection tag sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection tag received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.
[0042] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection tag, and the message carrying the detection tag includes a memory detection message and a service message carrying the detection tag.
[0043] In some possible implementations, the fault location module is specifically configured to:
[0044] The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received and the number of messages carrying the detection mark sent by the at least one switching node.
[0045] In some possible implementations, the detection configuration further includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.
[0046] In some possible implementations, the number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.
[0047] In a fourth aspect, the present application provides a detection controller. The detection controller includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions to cause the detection controller to perform the method described in any implementation of the first aspect of the present application.
[0048] In a fifth aspect, the present application provides a computing cluster. The computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes;
[0049] A first node in the computing cluster is configured to receive a detection configuration sent by a detection controller, wherein the detection configuration includes information about a second node detected by the first node, the first node and the second node being any computing node in the plurality of computing nodes or any switching node in the bus switching network;
[0050] The first node is further configured to send a message carrying a detection flag to the second node according to the detection configuration;
[0051] at least one switching node, configured to send to the detection controller a number of messages carrying the detection tag received by the at least one switching node and a number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection from the second node.
[0052] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a detection controller to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.
[0053] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing cluster to execute the fault location method described in the second aspect or any implementation of the second aspect.
[0054] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed on a detection controller, enables the detection controller to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.
[0055] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing cluster, enables the computing cluster to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.
[0056] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.
[0058] FIG1 is a schematic diagram of the architecture of a computing cluster provided in an embodiment of the present application;
[0059] FIG2 is a schematic diagram of a process of transmitting a memory access request by a switching chip in a bus switching network node according to an embodiment of the present application;
[0060] FIG3 is a schematic diagram of the architecture of a fault location system provided in an embodiment of the present application;
[0061] FIG4A is a flowchart of a fault location method provided in an embodiment of the present application;
[0062] FIG4B is a flowchart of another fault location method provided in an embodiment of the present application;
[0063] FIG5 is an interactive flow chart of a fault location method provided in an embodiment of the present application;
[0064] FIG6 is a schematic diagram of counting messages carrying detection tags according to an embodiment of the present application;
[0065] FIG7 is a schematic diagram of accessing a memory by reading according to an embodiment of the present application;
[0066] FIG8 is a schematic diagram of accessing memory by writing according to an embodiment of the present application;
[0067] FIG9 is a schematic diagram of a process for exception handling provided in an embodiment of the present application;
[0068] FIG10 is a schematic diagram of the life cycle of a detection task provided in an embodiment of the present application;
[0069] FIG11 is a schematic diagram of an application scenario of a fault location method provided in an embodiment of the present application;
[0070] FIG12 is a schematic diagram of a matching rule provided in an embodiment of the present application;
[0071] FIG13 is a schematic diagram of a method for determining a fault location according to an embodiment of the present application;
[0072] FIG14 is a schematic structural diagram of a detection controller provided in an embodiment of the present application;
[0073] FIG15 is a hardware structure diagram of a detection controller provided in an embodiment of the present application. DETAILED DESCRIPTION
[0074] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.
[0075] To facilitate understanding, some technical terms involved in this application are first introduced.
[0076] A bus switching network, also known as a bus network, is a network that implements data exchange via a bus. A bus switching network typically includes at least one switching node, such as a bus switch. Bus switching networks can employ various network architectures, including but not limited to the Clos network architecture and the Dragonfly network architecture. For ease of description, the following example uses a bus switching network employing the Clos network architecture.
[0077] As shown in Figure 1, a simple Clos network consists of the following switching layers: leaf switching layer and spine switching layer. The leaf switching layer includes access switches (also called leaf switches), which aggregate traffic from compute nodes (such as servers) and are directly connected to the spine switching layer. The spine switching layer includes spine switches, which are directly connected to leaf switches.
[0078] The leaf switching layer is located at the bottom layer of the bus switching network. Therefore, the leaf switches in the leaf switching layer are also called level 1 (L1) switches or L1 switching nodes. The spine switching layer is located at the upper layer of the bus switching network. Therefore, the spine switches in the spine switching layer are also called level 2 (L2) switches or L2 switching nodes. In the example of Figure 1, the bus switching network can include leaf switches L1-1 to L1-7 and spine switches L2-1 to L2-8.
[0079] The bus switching network can connect different computing nodes to form a computing cluster (also known as a cluster computing system or super node), thereby increasing computing power through horizontal expansion. Still using Figure 1 as an example, the computing node includes a processor, which may include but is not limited to a central processing unit (CPU) and a neural processing unit (NPU). The CPU / NPU in the computing node can achieve memory access between computing nodes (cross-node memory access) through the bus switching network.
[0080] The following describes how to implement memory access between computing nodes by switching nodes.
[0081] A switching node may include one or more switching chips that support data exchange via a bus. As shown in Figure 2, a switching node may include the following switching chips: switching chip-1, switching chip-2, and switching chip-3. When CPU 201 accesses memory data at location 0x3FE0 of NPU 202 (i.e., memory address 0x3FE0), CPU 201 sends a data request command to the adjacent switching chip-1. Switching chip-1 queries the address table based on memory address 0x3FE0, obtains the network identifier (NID) 127 corresponding to the memory address, converts the data request command into a data request message, sets the network address NID to 127 in the data request message, then queries the routing table to obtain the outgoing interface 16, and forwards it to switching chip-2. After the data request message is forwarded to switching chip-3, switching chip-3 restores the data request message to a data request command and transmits it to NPU 202, thereby completing the transmission process of a memory access request.
[0082] As the scale of computing clusters (e.g., server chip size and bus size) continues to grow, the number of switching chips and chip interfaces traversed by a single cross-node memory access increases dramatically, increasing the likelihood of bus switching network transmission failures. When a bus switching network fails, locating the fault location as quickly as possible to improve cluster service availability becomes a pressing issue. Widely used fault location tools in the industry include traceroute, ping, or pingmesh. However, most of these tools are built for Internet Protocol (IP) services and use media access control (MAC) addresses or IP addresses for addressing. Bus switching networks typically use memory addresses for addressing memory accesses. Therefore, fault location tools such as traceroute, ping, or pingmesh are unable to meet the fault location requirements for memory accesses over bus switching networks.
[0083] In view of this, the present application provides a fault location method. The method can be performed by a detection controller. The detection controller is also called a super node controller, which is used to locate the fault when the bus switching network in the super node (i.e., computing cluster) fails. Among them, the detection controller can be software, and the software can be deployed on the computing node, switching node in the computing cluster, or deployed on an independent node. The computing node, switching node or independent node executes the program code of the software, thereby executing the fault location method of the present application. The detection controller can also be hardware. For example, the detection controller can be a server with a fault location function, and the server executes the fault location method of the present application when it is running.
[0084] Specifically, a detection controller can issue a detection configuration to a first node in a computing cluster. The detection configuration includes information about a second node that is detected by the first node. The computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes. The first node and the second node can be any computing node in the multiple computing nodes or any switching node in the bus switching network. The detection controller can then obtain the number of messages carrying a detection tag (trace_tag) received by at least one switching node and the number of messages carrying the detection tag sent by at least one switching node. The at least one switching node is a switching node in the bus switching network through which the first node initiates detection from the second node. The messages carrying the detection tag are sent by the first node according to the detection configuration. The detection controller then determines the fault location based on the number of messages carrying the detection tag received by at least one switching node and the number of messages carrying the detection tag sent by at least one switching node.
[0085] This method determines packet loss of probe-tagged packets at at least one switching node in a bus switching network based on the number of probe-tagged packets received and sent by that node, thereby locating faults in the bus switching network. Even if the bus switching network uses per-packet load balancing, resulting in uncertain, unpredictable, and non-traceable forwarding paths, the method can still locate packet loss and thus fault location, meeting service requirements.
[0086] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the fault location system of the embodiment of the present application is first introduced below with reference to the accompanying drawings.
[0087] Referring to FIG3 , which illustrates the architecture of a fault location system, fault location system 300 includes a detection controller 302. Furthermore, fault location system 300 may also include a detection agent 304 and a statistical agent 306. Detection agent 304 may be deployed on a compute node or switch node in a computing cluster, while statistical agent 306 may be deployed on a switch node. Similar to detection controller 302, detection agent 304 and statistical agent 306 may be software. Nodes (e.g., compute nodes or switch nodes) on which such software is deployed execute the software's program code, thereby executing the steps of the fault location method performed by detection agent 304 and statistical agent 306.
[0088] Specifically, the detection controller 302 is used to send a detection configuration to the first node in the computing cluster, and the detection configuration includes information about the second node that the first node initiates detection. The first node and the second node can be any computing node among a plurality of computing nodes or any switching node in a bus switching network. In some embodiments, the detection controller 302 can send a detection configuration to all nodes in the computing cluster (including the first node and the second node), and nodes of different role types (such as computing nodes and switching nodes) use different information in the detection configuration. For ease of description, this application takes the first node as an example of a computing node.
[0089] Among them, the detection controller 302 can send a detection task to the first node, and the detection task includes a detection configuration. In some examples, the detection controller 202 can also send a detection configuration to the first node, and the first node generates a detection task according to the detection configuration. The detection task can be performed when the user service is suspended. In other words, the detection task can be executed when the user service is suspended. In this case, the messages sent from the first node are all memory detection messages. The detection task can also be performed when the user service is running. In other words, the detection task can be executed when the user service is running, the detection mode is global detection, and the messages sent from the first node include memory detection messages and service messages carrying detection tags.
[0090] The detection agent 304 deployed in the first node is used to send a message carrying a detection tag according to the detection configuration. The message carrying the detection tag is used to detect connectivity by accessing memory across nodes. The message carrying the detection tag sent by the detection agent 304 can be a request message. The source node of the request message is the first node, and the destination node of the request message is the second node. The detection agent 304 on the second node is allocated a memory space (detection memory) for access by the detection agent 304 of other nodes (such as the first node) to detect connectivity. The second node can detect connectivity after receiving the request message and return a response message corresponding to the request message. It should be noted that the response message corresponding to the request message can also carry a detection tag. The detection tag can be set to a target value, for example, trace_tag can be set to 1, so that the switching node can recognize the detection tag and then perform message statistics. A message carrying a detection tag can cover a load sharing path and a routing table entry. Multiple messages carrying detection tags generated according to the detection configuration can traverse all load sharing paths and all routing table entries during the forwarding process of the bus switching network, so as to comprehensively detect problems in the bus switching network.
[0091] The statistics agent 306 deployed in at least one switching node is configured to count the number of messages carrying a detection tag received and sent by the at least one switching node. When a response message carries a detection tag, the message carrying the detection tag includes a request message and a response message corresponding to the request message; when a response message does not carry a detection tag, the message carrying the detection tag includes the request message.
[0092] In some possible implementations, when the detection mode is global detection, the service messages of the computing nodes that enable the detection function also carry a detection tag. For example, when the first node is a computing node, the service messages of the first node also carry a detection tag. Accordingly, the messages carrying the detection tag include memory detection messages (such as memory detection request messages, or memory detection request messages and memory detection response messages) and service messages carrying the detection tag. In other words, when the statistical agent 306 counts the number of messages carrying the detection tag received by the switching node and the number of messages carrying the detection tag sent by the switching node, it can count the number of memory detection messages and service messages carrying the detection tag received by the switching node and the number of memory detection messages and service messages carrying the detection tag sent by the switching node. Among them, the statistical agent 306 can count the number of messages carrying the detection tag received at the input interface of the port of the switching node, and count the number of messages carrying the detection tag sent at the output interface of the port of the switching node.
[0093] The detection controller 302 is further configured to obtain the number of messages carrying the detection tag received and sent by at least one switching node, and determine the fault location based on the number of messages carrying the detection tag received and sent by the at least one switching node. The at least one switching node is a switching node in the bus switching network through which the first node initiates a detection from the second node.
[0094] It should be noted that the statistics agent 306 can periodically report statistical data to the detection controller 302. The statistical data includes the number of detection-tagged packets received and sent by the switching node. The reporting method can be active or passive in response to a statistics collection request from the detection controller 302. The detection controller 302 obtains the statistical data reported by the statistics agent 306, thereby obtaining the number of detection-tagged packets received and sent by at least one switching node.
[0095] In some possible implementations, the detection controller 302 may also receive a fault record reported by the first node or the second node, where the fault record is determined by the first node or the second node based on the expected detection result and the actual detection result. Accordingly, upon receiving the fault record, the detection controller 302 may trigger a fault location process to determine the fault location. For example, upon receiving the fault record reported by the first node or the second node, the detection controller 302 may determine the fault location in the bus switching network based on the number of messages carrying the detection tag received and the number of messages carrying the detection tag sent by at least one switching node.
[0096] It should be noted that the detection controller 302 can determine the fault location based on the fault record. For example, the detection controller 302 can obtain the source node and destination node from the fault record, and obtain the number of packets carrying the detection tag received and sent by the switching node on the path from the source node to the destination node (for example, a load sharing path). The fault location is determined based on this number, thereby reducing the detection range and improving detection efficiency.
[0097] Figure 3 illustrates an example of a fault record reported by a destination node being received by a detection controller 302, and does not limit the technical solution of this application. Furthermore, Figure 3 also illustrates control flows (e.g., those for issuing detection configurations, collecting statistical data, or fault records) and detection flows (e.g., those for data messages carrying detection flags) using different line types. The control flows can be transmitted via the control plane, while the data flows can be transmitted via the data plane.
[0098] Next, the fault location method according to the embodiment of the present application will be introduced with reference to the accompanying drawings.
[0099] 4A shows a flow chart of a fault location method, which is applied to the detection controller 302 and includes:
[0100] S402: The detection controller 302 sends a detection configuration to the first node in the computing cluster.
[0101] A computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes. A detection configuration includes information about a second node initiated by a first node. The first node and the second node can be any computing node among the multiple computing nodes or any switching node in the bus switching network. The first node can include one or more nodes, and the second node can include one or more nodes. The path from the first node to the second node includes a forwarding path of the bus switching network. For example, the path from the first node to the second node can cover all forwarding paths of the bus switching network, so that all forwarding paths can be traversed subsequently to locate the fault.
[0102] In some possible implementations, a computing cluster includes multiple node groups, and different computing nodes within the same node group are directly connected to the same switch node. Accordingly, the second node may include a computing node at the same location in a node group other than the node group to which the first node belongs. In this way, the path from the first node to the second node can overlap the forwarding path of the bus switching network.
[0103] Specifically, when the detection controller 302 sends the detection configuration to the first node, it can send a detection task, and the detection task includes the above-mentioned detection configuration. Alternatively, the detection controller 302 can directly send the detection configuration so that the first node generates a detection task according to the detection configuration. The information of the second node included in the detection configuration can be represented by a node list. In some examples, the node list can record the identifier of the second node. This embodiment uses a memory access method to locate the fault. Based on this, the detection configuration can also include an expected detection result. Based on the expected detection result and the actual detection result, it can be determined whether a fault occurs. If so, fault location can be triggered.
[0104] Furthermore, the detection configuration may also include a detection access mode. When the detection access mode is read access, the first node (source node) can determine whether the bus switching network has failed based on the expected detection result and the actual detection result. When the detection access mode is write access, the second node (destination node) can determine whether the bus switching network has failed based on the expected detection result and the actual detection result. The detection configuration may also include a detection access memory address to facilitate reading data from or writing data to the corresponding memory address, thereby detecting connectivity. Considering that the detection task can be a periodic task, the detection configuration may also include at least one of the following: detection frequency, detection time interval, number of detections, detection start time, and detection end time. To facilitate fine-grained packet counting, the detection configuration may also indicate a path number. The path number may be the number of one or more paths. Multiple paths may be equal-cost paths, for example, multiple paths with the same source node and destination node. In this way, when counting packets, separate statistics can be performed for packets with different path numbers.
[0105] S404: The detection controller 302 obtains the number of messages carrying the detection flag received by at least one switching node and the number of messages carrying the detection flag sent by at least one switching node.
[0106] Messages carrying a detection flag are used to detect connectivity by accessing memory across nodes. Messages carrying a detection flag include request messages. The source node of the request message is the first node, and the destination node of the request message is the second node. In some possible implementations, a response message corresponding to a request message carrying a detection flag may also carry a detection flag. The source node of the response message is the second node, and the destination node of the response message is the first node.
[0107] A message carrying a detection tag (e.g., a request message carrying a detection tag) can be generated by the first node based on a detection configuration. Specifically, upon receiving a detection task including the detection configuration, the first node can generate a request message carrying a detection tag in response to the detection task. Alternatively, upon receiving the detection configuration, the first node can generate a detection task and then generate a request message carrying a detection tag in response to the detection task. The first node can then send the request message carrying the detection tag to the second node. At least one switching node in the bus switching network can forward the request message to the second node. The at least one switching node can be the switching node in the bus switching network through which the first node initiates a detection from the second node.
[0108] In order to ensure that the message carrying the detection mark (such as the request message) can cover as many load sharing paths as possible, the first node can send multiple messages carrying the detection mark to the second node (for example, each node in the second node). In some examples, for any node in the second node, the number of messages carrying the detection mark (such as the request message) sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node (for example, load sharing paths). Assuming that there are M load sharing paths from the first node to the second node, at least N detection requests need to be sent for each detection, where N>=M. In addition, in order to ensure that data can be obtained from other nodes each time it is accessed, the detection memory of the second node can usually be configured as non-cacheable to avoid hitting data in the cache and affecting the detection results.
[0109] Upon receiving the request message, the second node may identify the detection tag in the request message. If the request message includes the detection tag, the second node may access the detection memory of the second node to detect connectivity. Based on the detection results, the second node may generate a response message carrying the detection tag and return the response message carrying the detection tag to the first node. At least one switching node in the bus switching network may forward the response message to the first node.
[0110] The number of detection-tagged messages received by a switching node is determined by identifying the detection tag in the received messages, and the number of detection-tagged messages sent by the switching node is determined by identifying the detection tag in the sent messages. Since request and response messages carrying the detection tag pass through the switching node, the statistics agent 306 deployed in the switching node can determine the number of detection-tagged messages received by identifying the detection tag in the received messages, and can determine the number of detection-tagged messages sent by identifying the detection tag in the sent messages. It should be noted that the above numbers can also be determined by other devices or equipment, and this embodiment does not limit this.
[0111] Furthermore, the packets carrying the detection flag counted by the statistics agent 306 deployed in the switching node may be different depending on the timing of executing the detection task.
[0112] When the detection task is executed when the user service is suspended, all messages sent from the first node are memory detection messages. Memory detection messages are messages dedicated to memory detection. In this case, the messages carrying the detection flag counted by the statistics agent 306 include memory detection messages.
[0113] When the detection task is executed during user service operation, indicating that the detection mode is global detection, the service messages of the detection-enabled computing nodes also carry the detection tag. Based on this, when the first node is any computing node among multiple computing nodes, the messages sent from the first node include memory detection messages and service messages carrying the detection tag. Messages carrying the detection tag include memory detection messages and service messages carrying the detection tag. Accordingly, the detection-tagged messages counted by the statistics agent 306 include memory detection messages and the aforementioned service messages carrying the detection tag.
[0114] S406: The detection controller 302 determines the fault location according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.
[0115] The detection controller 302 can perform node-granular analysis to determine the fault location. Specifically, the detection controller 302 can check whether the number of detection-tagged messages received by and sent by the same switching node in at least one switching node is equal. If the number of detection-tagged messages received by a first switching node in at least one switching node is not equal to the number of detection-tagged messages sent by the first switching node, the detection controller 302 determines that the fault location in the bus switching network includes the first switching node.
[0116] The detection controller 302 can also analyze from an upstream and downstream perspective to determine the fault location. Specifically, the bus switching network includes a first switching node and a second switching node, and the outbound interface of the first switching node is equal to the inbound interface of the second switching node. Therefore, the first switching node can be considered an upstream node of the second switching node, and the second switching node can be considered a downstream node of the first switching node. The detection controller 302 can check whether the number of packets carrying the detection tag sent by the first outbound interface of the first switching node is equal to the number of packets carrying the detection tag received by the first inbound interface of the second switching node. When the number of packets carrying the detection tag sent by the first switching node through the outbound interface is not equal to the number of packets carrying the detection tag received by the second switching node through the inbound interface, the detection controller 302 determines that the fault location in the bus switching network includes the link from the outbound interface of the first switching node to the inbound interface of the second switching node.
[0117] Among them, the link from the outbound interface of the first switching node to the inbound interface of the second switching node may include a network transmission channel from the outbound interface of the first switching node to the inbound interface of the second switching node, or the outbound interface of the first switching node and the inbound interface of the second switching node. In other words, when a failure occurs in the link from the outbound interface of the first switching node to the inbound interface of the second switching node, it may be a failure of the outbound interface of the first switching node, a failure of the inbound interface of the second switching node, or a failure of the network transmission channel.
[0118] Based on the foregoing description, the fault location method of the present application analyzes the packet loss of detection-tagged messages in at least one switching node in a bus switching network based on the number of detection-tagged messages received and sent by at least one switching node, thereby locating faults in the bus switching network. Even if the bus switching network uses a packet-by-packet load balancing method, resulting in an uncertain, unpredictable, and non-traceable forwarding path, the location of packet loss can still be determined, thereby locating the fault and meeting service requirements.
[0119] 4A introduces the fault location method of the present application from the perspective of the detection controller 302. Next, the fault location method of the present application is introduced from the perspective of a computing cluster.
[0120] 4B shows a flow chart of a fault location method. The method is applied to a computing cluster, wherein the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes. The method includes:
[0121] S403 : The first node in the computing cluster receives the detection configuration sent by the detection controller 302 .
[0122] The detection configuration includes information about the second node detected by the first node. The first node and the second node can be any computing node among a plurality of computing nodes or any switching node in a bus switching network.
[0123] S405: The first node sends a message carrying a detection flag to the second node according to the detection configuration.
[0124] When the detection controller 302 sends down the detection configuration in the form of carrying the detection configuration in the detection task, the first node can respond to the detection task, generate a message carrying the detection tag according to the detection configuration, such as a request message carrying the detection tag, and then send the message carrying the detection tag to the second node.
[0125] When the detection controller 302 directly sends down the detection configuration, the first node may generate a detection task according to the detection configuration, and then generate a message carrying a detection tag according to the detection configuration in response to the detection task, and then send the message carrying the detection tag to the second node.
[0126] The first node includes a detection agent. The first node can send a message carrying a detection flag through the detection agent. For example, the detection agent in the first node sends a message carrying a detection flag to the second node according to the detection configuration.
[0127] In some possible implementations, the first node is any computing node among multiple computing nodes. The detection configuration may also indicate a detection mode. When the detection mode is global detection, the first node carries a detection tag in the service message. Accordingly, the message carrying the detection tag includes a memory detection message and a service message carrying the detection tag. In this way, fault detection can be performed while the service is running without having to wait for the service to be suspended. Moreover, carrying the detection tag in the service message for detection can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and achieve resource reuse.
[0128] It should also be noted that, in order to ensure that the messages cover as many paths as possible, the number of messages carrying the detection tag sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node. Assuming that the number of paths (e.g., load sharing paths) from the first node to the second node is M, where M is a positive integer, the first node can send N messages carrying the detection tag to the second node based on the detection configuration, where N is greater than or equal to M.
[0129] Furthermore, the second node can also perform a memory probe based on the message carrying the probe tag to obtain the actual probe result. When the probe access mode is read, the second node can also return the actual probe result. In this way, both the first and second nodes can obtain the actual probe result. When the first or second node detects a discrepancy between the expected probe result and the actual probe result, it can report a fault record to the probe controller 302. The fault record is used to trigger the probe controller to locate the fault.
[0130] S407 : At least one switching node in the bus switching network included in the computing cluster sends to the detection controller 302 the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node.
[0131] At least one switching node is a switching node in the bus switching network through which the first node initiates a probe from the second node. The switching node includes a statistical agent. Accordingly, the switching node can use the statistical agent to count and report the number of messages carrying the probe tag received and sent by the at least one switching node to the probe controller 302. In this manner, the probe controller 302 can perform fault location based on the number of messages carrying the probe tag received and sent by the switching node.
[0132] Based on the above description, this method reports the number of detection-tagged packets received and sent by at least one switching node in the bus switching network to the detection controller 302. The detection controller 302 can then analyze packet loss among the detection-tagged packets in the switching node to locate faults in the bus switching network. Even if the bus switching network uses per-packet load balancing, resulting in uncertain, unpredictable, and non-traceable forwarding paths, the location of packet loss can still be determined, thereby locating the fault and meeting service requirements.
[0133] The detection controller 302 is the organizer of the detection task and is responsible for coordinating the computing nodes (e.g., the detection agent 304 deployed in the computing nodes) and the switching nodes (e.g., the statistics agent 306 deployed in the switching nodes) to jointly complete the detection task. The following describes the fault location method of the present application from the perspective of the interaction between the detection controller 302, the computing nodes (or the detection agent 304 deployed in the computing nodes), and the switching nodes (the statistics agent 306 deployed in the switching nodes).
[0134] Referring to FIG5 , an interactive flow chart of a fault location method is shown. The method includes:
[0135] S502: The detection controller 302 issues a detection task.
[0136] A detection task refers to a task of generating a message carrying a detection tag and transmitting the message through a bus switching network to perform connectivity detection. The detection task can be issued by the detection controller 302, or the detection controller 302 can issue a detection configuration (for example, a detection task description) to the first computing node, and the first computing node generates it according to the detection configuration. Among them, the detection configuration may include information of a second computing node (detection request target node) that is detected by the first computing node (also referred to as a detection execution node or a detection request source node). In some examples, the detection task can cover the interface input and output directions of the switching node (for example, all interface input and output directions), as well as the forwarding table entries of the switching node (for example, all forwarding table entries).
[0137] In some possible implementations, the probe configuration further includes at least one of a probe frequency, a probe count, a probe start time, a probe end time, a probe rate (i.e., a probe message sending rate), and a probe time interval. The probe frequency, the probe count, the probe start time, the probe end time, the probe rate, and the probe time interval may indicate a probe task repetition method, and the first computing node may independently and repeatedly initiate a probe task based on the aforementioned parameters in the probe configuration. For example, the first computing node may periodically initiate a probe task based on the probe time interval.
[0138] Furthermore, the detection configuration may also include a detection expected result. If the detection access mode is a read mode, the detection expected result may be the data requested to be read. If the detection access mode is a write mode, the detection expected result may be the detection request content, such as the data to be written.
[0139] It should be noted that FIG5 illustrates an example in which the detection controller 302 directly issues a detection task. In other possible implementations of the embodiment of the present application, the detection controller 302 may also issue a detection configuration, and the first computing node generates a detection task according to the detection configuration.
[0140] S504 : The detection agent device 304 deployed in the first computing node sends a request message carrying a detection tag in response to the detection task.
[0141] The detection agent 304 can be deployed on a computing node. The detection agent 304 can be a computing task execution engine of the computing node, or it can be deployed on the computing node in the form of a virtual machine (VM), and the computing task runs in the VM. The detection agent 304 can first allocate a block of memory for access by the detection agent 304 deployed in other nodes, and set the memory data to an agreed value. The memory space allocated by the detection agent 304 must be large enough so that all interleaved channels can be covered when the detection agent 304 of other nodes accesses it. It should be noted that if the detection access is implemented in a write manner, the detection agent 304 can allocate multiple blocks of memory for detection access by other nodes.
[0142] Specifically, the detection agent 304 deployed in the first computing node can carry a detection tag trace_tag in the message, send several request messages carrying the detection tag to other nodes according to the detection task, wait for the response message carrying the detection tag, and verify the detection result. Depending on the detection access mode, the request message type can be different. For example, when the detection access mode is a read mode, the request message can be a load (LD)-based message or a direct memory access (DMA) READ-based message. When the detection access mode is a write mode, the request message can be a store (ST)-based message or a DMA WRITE-based message. Among them, LD and ST instructions are CPU memory instructions. The LD instruction is used to read a single data from a memory address, which can usually be 8 to 256 bits. The ST instruction is used to write a single data (for example, 8 to 256 bits) to a memory address. DMA READ / DMA WRITE can be used for reading and writing large segments of data, such as reading and writing data of more than 64 bytes.
[0143] Furthermore, to ensure that the request message carrying the detection tag can cover as many load-sharing paths as possible, the first computing node can send multiple request messages carrying the detection tag to the second computing node. Assuming that there are M paths (for example, load-sharing paths) from the first computing node to the second computing node, the first computing node can send at least N request messages carrying the detection tag each time it detects, where N>=M. In addition, to ensure that data can be obtained from other nodes each time it is accessed, the detection memory of the second computing node can usually be configured as non-cacheable to avoid hitting data in the cache and affecting the detection results.
[0144] When the detection task indicates a detection start time, the detection agent 304 deployed in the first computing node can also set a timer, and when the time set by the timer arrives, it sends a message carrying a detection tag to request cross-node access to memory and detect connectivity. When the detection task indicates a detection end time, the detection agent 304 deployed in the first computing node can also stop sending messages carrying the detection tag when the detection end time arrives. For example, when global detection is turned on, the detection agent 304 deployed in the first computing node can also stop sending memory detection messages when the detection end time arrives, turning off the function of carrying the detection tag in the business message.
[0145] S506: The statistics agent 306 deployed in the switching node triggers message statistics.
[0146] The statistics agent 306 can trigger statistics on the number of sent packets carrying a detection tag and the number of received packets carrying a detection tag. Specifically, the statistics agent 306 can initialize the number of sent packets carrying a detection tag to 0 and the number of received packets carrying a detection tag to 0. When the outbound interface of the switching node sends a packet carrying a detection tag, the statistics agent 306 updates the number of sent packets carrying a detection tag, for example, by increasing the number of sent packets carrying a detection tag by 1. When the inbound interface of the switching node receives a packet carrying a detection tag, the statistics agent 306 updates the number of received packets carrying a detection tag, for example, by increasing the number of received packets carrying a detection tag by 1. It should be noted that the statistics agent 306 can count the number of packets passing through each interface separately, or count the number of sent packets carrying a detection tag and the number of received packets carrying a detection tag by the switching node regardless of the interface.
[0147] The detection mark may include various forms, and the following describes different forms of the detection mark with examples.
[0148] In a first possible implementation, the detection flag may be a bit in the message used to identify whether statistics are required. When the value of the bit is a set value (eg, 1), it indicates that the message needs to be counted.
[0149] In a second possible implementation, the detection tag can be multiple bits in the message, and the multiple bits can have multiple values. Among all possible values of the detection tag, some values are used to identify the message as a message to be counted, and the remaining values are used to identify the message as a message not to be counted. For example, when the detection tag includes two bits, all possible values of the detection tag are: 0 (binary representation is 00), 1 (binary representation is 01), 2 (binary representation is 10), 3 (binary representation is 11), among which the value 0 is used to identify the message as a message not to be counted, and the values 1, 2, and 3 are used to identify the message as a message to be counted. Furthermore, the value that identifies the message as a message to be counted can be further subdivided, for example, the value is also used to identify the category of the message to be counted. For example, the messages to be counted can be classified according to the source node, and some values of the detection tag can be used to identify the messages to be counted of each source node.
[0150] In a third possible implementation, the detection flag may be multiple bits in a message, wherein one bit is used to identify whether the message is a message to be counted, and the remaining bits are used to identify the category of the message to be counted. Still taking the example of a detection flag including two bits, the first bit identifies whether the message is a message to be counted, and the second bit identifies the category of the message to be counted. For example, the second bit may identify the source node of the message to be counted.
[0151] The statistical agent 306 is deployed on the switching node. Based on this, the statistical agent 306 can first perform statistical matching content configuration on the switching node. The statistical matching content includes but is not limited to matching fields and matching feature values. For example, the statistical agent 306 can be configured: the matching field is the trace_tag field, and the matching feature value is 1, which is used to identify messages with trace_tag=1. Referring to Figure 6, when a message is forwarded on the switching node, the statistical agent 306 performs feature matching. If the field value of the matching field of the message meets the matching feature value, for example, if trace_tag=1 in the message, then the message is determined to be a message carrying a detection tag. The statistical agent 306 performs statistics on the message in the inbound and outbound directions respectively.
[0152] S508: The detection agent 304 deployed in the second computing node receives the request message carrying the detection tag forwarded by the switch node, and performs memory detection.
[0153] S509: The detection agent 304 deployed in the second computing node generates a response message and returns the response message to the first computing node.
[0154] The detection agent 304 deployed in the second computing node can perform memory detection in a read mode or a write mode, which will be described below with reference to examples.
[0155] Referring to FIG7 , a schematic diagram of memory detection by reading is shown. The first computing node (source node) uses a read method, such as LD or DMA READ, to read data from the detection memory of the second computing node (destination node), wherein the detection memory of the second computing node is non-cacheable memory. In this example, the actual detection result can be the data read from the detection memory. When the response message carrying the above data is forwarded to the first computing node by switching nodes 1, 2, and 3 (denoted as SW1, SW2, and SW3) in the bus switching network, the first computing node can obtain the data (or value) of the memory address in the detection memory of the second computing node.
[0156] Referring to FIG8 , a schematic diagram of memory detection by writing is shown. The first computing node (source node) uses a writing method, such as ST or DMA WRITE, to write data to the detection memory of the second computing node (destination node), wherein the detection memory of the second computing node is a non-cacheable memory. In this example, the actual detection result can be the data written to the detection memory of the second computing node. The second computing node can also detect whether the detection memory is rewritten to the expected detection result (pre-agreed data) to determine whether a fault has occurred. If a block of memory is not rewritten successfully, it indicates that a bus switching network has failed. Using FIG8 as an example, when the switching node 2 (such as SW2) in the bus switching network fails, it may cause a block of memory in the detection memory of the second computing node to fail to be rewritten successfully. The second computing node detects that a block of memory has failed to be rewritten and determines that the bus switching network has failed.
[0157] If the request message carrying the detection mark or the response message corresponding to the request message is discarded in the bus switching network, it may cause a memory access timeout and generate a memory access exception, which may be, for example, a synchronous external abort (SEA) exception. Normally, after the memory access exception is taken over by the operating system (for example, the operating system kernel), the operating system can end the detection task to eliminate the memory access exception. In order to ensure that the detection task can continue, the detection agent device 304 can register an exception handler (for example, an exception handling function) in advance. Through the jump of the exception handler, the detection task can be jumped from the kernel space of the operating system back to the user space. In this way, the detection agent device 304 can continue to execute the detection task.
[0158] Specifically, the detection agent 304 can set an exception jump point during the execution of the detection task, and can save the context of the detection task at the exception jump point so that the execution of the detection task can be resumed based on the saved context. When a memory access exception occurs, the detection agent 304 can enter the exception handler, and the exception handler can return to the exception jump point through a function with cross-function jump capability.
[0159] Among them, setting the exception jump point and realizing cross-function jump can be realized by setjmp function and longjmp function. Referring to the flowchart of an exception handling shown in Figure 9, the detection agent device 304 includes a detection task execution engine, the detection task execution engine registers the SEA exception handler, and then obtains the memory address indicated by the detection task, executes the setjmp function, and saves the context of the function here in the jmp_buf structure for longjmp to restore the context from this structure. If the function is called directly, the return value is 0; if the function returns from the longjmp call, the return value is non-zero, which is provided by the longjmp function. According to the return value of the function, the detection task execution engine can determine whether the setjmp function call is a direct call or a jump from elsewhere.
[0160] When the setjmp function call is a direct call, the detection task execution engine can initiate an access to the memory address and wait for a response. If the request or response message is discarded, a SEA exception occurs. The detection task execution engine enters the exception handler and performs exception handling, including recording the access object with the error, restoring the context through longjmp, returning to the setjmp function, and then reporting the fault record to the detection controller 302. After completing the exception handling, the detection task execution engine can continue with the remaining detection tasks.
[0161] S510 : The detection agent 304 deployed in the first computing node receives the response message forwarded by the switching node and records the actual detection result.
[0162] It should be noted that FIG5 illustrates an example of a read access mode. When the write access mode is used, the fault location access according to the embodiment of the present application can be performed without executing S509 and S510. For example, if the actual detection result is inconsistent with the expected detection result, the second computing node can directly report the fault record.
[0163] S512: When the detection agent 304 deployed in the first computing node determines that a fault has occurred according to the actual detection result, it reports a fault record.
[0164] When accessing the memory in a read manner, the detection agent device 304 deployed in the first computing node can determine whether a fault has occurred by checking whether the actual detection result read is consistent with the expected detection result. If not, it indicates that a fault has occurred, and the first computing node can report the fault record.
[0165] When the memory is detected in a write manner, the detection agent device 304 deployed in the second computing node can check whether the detection memory of the second computing node is rewritten to the expected detection result. If the detection memory is not rewritten successfully, it indicates that a fault has occurred, and the second computing node can report the fault record to the detection controller 302.
[0166] Among them, when the detection task is a repetitive task, for example, a periodic task, the detection agent device 304 deployed in the second computing node can periodically erase the detection memory. After each detection cycle, it checks whether the detection memory of the second computing node is rewritten to the expected detection result, thereby determining whether a fault occurs.
[0167] S514 , the detection controller 302 collects statistical data from the statistical agent 306 .
[0168] The detection controller 302 can proactively collect statistical data from the switching node (e.g., a statistics agent 306 deployed in the switching node). In some examples, the statistics agent 306 can also proactively report statistical data to the detection controller 302. The statistical data can include the number of packets carrying the detection tag received by the switching node and the number of packets carrying the detection tag sent by the switching node.
[0169] As shown in Figure 10, the entire life cycle of a detection task can be divided into a memory detection phase, a statistics collection phase, and a data verification phase. After the memory detection phase, the detection controller 302 can collect statistical data to ensure that all request messages carrying the detection flag have been sent to the destination under the correct circumstances, ensuring that the statistical data collection is complete.
[0170] Specifically, when the memory detection phase ends, the statistical agent 306 can wait for a period of time, which may include the time required for the request message carrying the detection mark sent by the first computing node to reach the second computing node through the bus switching network. Furthermore, the time may also include the time required for the response message sent by the second computing node to reach the first computing node through the bus switching network. In this way, under normal circumstances, the request message and the response message have arrived at the destination node, and the statistical agent 306 can start collecting statistical data.
[0171] S516 : The detection controller 302 determines the fault location based on the statistical data.
[0172] The detection controller 302 can trigger fault location when a fault is detected by the detection task. For example, when the detection controller 302 receives a fault record reported by a computing node (such as the first computing node), it can determine the fault location based on statistical data. As shown in Figure 10, the detection controller 302 can verify the data based on the statistical data after collecting it, thereby achieving fault location. The detection controller 302 can perform data verification in at least one of the following ways:
[0173] In a first implementation, the detection controller 302 checks whether the number of detection-tagged messages received by and sent from the same switching node is equal. If the number of detection-tagged messages received by a first switching node among at least one switching node is not equal to the number of detection-tagged messages sent by the first switching node, the detection controller 302 determines that the fault location in the bus switching network includes the first switching node.
[0174] In a second implementation, a bus switching network includes a first switching node and a second switching node, each of which includes at least one input interface and at least one output interface. The output interface of the first switching node is connected to the input interface of the second switching node. The detection controller 302 checks whether the number of packets carrying a detection tag sent by the output interface of the first switching node is equal to the number of packets carrying a detection tag received by the input interface of the second switching node. When the number of packets carrying a detection tag sent by the output interface of the first switching node is not equal to the number of packets carrying a detection tag received by the input interface of the second switching node, the detection controller 302 determines that the fault location in the bus switching network includes a link from the output interface of the first switching node to the input interface of the second switching node. The link from the output interface of the first switching node to the input interface of the second switching node may include a network transmission channel from the output interface of the first switching node to the input interface of the second switching node, or a link between the output interface of the first switching node and the input interface of the second switching node.
[0175] When the detection task is a repetitive task, such as a periodic task, the detection controller 302 may subtract the statistical data of the current cycle from the statistical data collected in the previous cycle (previous round) to obtain the incremental data of the current cycle, as shown below: n =C tn -C tn-1 (1)
[0176] Among them, C tn Indicates the statistics of the current period, C tn-1 Indicates the statistical data of the previous period, C n Indicates the incremental data of the current period.
[0177] Then, the detection controller 302 may perform verification based on the incremental data of the current cycle. Specific verification methods may include:
[0178] In the first verification method, the detection controller 302 obtains the number of packets carrying the detection mark received by all interfaces of the switching node according to the incremental data of the current cycle (denoted as ∑ i I i ), and the number of packets carrying detection tags sent by all interfaces of the switching node (denoted as ∑ i O i ), and then determine whether the following formula is satisfied: ∑ i I i =∑ i O i (2)
[0179] If not, it means that the switching node has a fault (or anomaly).
[0180] In the second verification method, assuming that the first switching node is connected to the interface b of the second switching node through the interface a, the detection controller 302 can obtain the number of packets carrying the detection mark sent by the interface a of the first switching node (denoted as S) according to the incremental data of the current cycle. out ), and the number of packets carrying the detection tag received by the interface b of the second switching node (denoted as T in ), and then determine whether the following formula is satisfied: S out =T in (3)
[0181] If not satisfied, for example, S out >T in , it means that there is a fault (or abnormality) in the link from interface a of the first switching node to interface b of the second switching node.
[0182] In some possible implementations, the detection controller 302 may also obtain expected statistical data for each interface of the switching node (for example, the expected number of packets sent or received carrying a detection tag, which may generally be equal to the number of packets sent by the source node carrying the detection tag), and then compare the collected statistical data with the actual data to determine the fault location. When determining the fault location, the detection controller 302 may also consider the success or failure of the detection task to improve the efficiency and accuracy of fault location.
[0183] Next, the fault location method of the present application is described in conjunction with specific application scenarios.
[0184] Referring to the schematic diagram of an application scenario of a fault location method shown in FIG11 , in this example, the detection controller issues a detection task to each computing node in the computing cluster. The detection task instructs each computing node to send a request message carrying a detection tag at a configured detection rate (detection message sending rate). Among them, the detection task also instructs each computing node in each node group to send a request message to the computing node at the same position in other node groups to access the detection memory of the computing node at the same position in other node groups, so as to cover all forwarding paths of the bus switching network and the routing table of the switching node in the bus switching network. That is, the first node can be each computing node in the computing cluster, and the second node can be a computing node at the same position in other node groups outside the node group where each computing node is located.
[0185] During the memory detection phase, compute nodes enable detection, and each compute node sends at least (Y-1) requests to cover all forwarding paths and the routing tables of the switch nodes in the bus switching network. Y is the number of node groups in the compute cluster. Furthermore, because switch nodes select random paths during forwarding, compute nodes can send as many request messages as possible. For example, each compute node can send N*(Y-1) request messages, thus ensuring maximum coverage of all paths.
[0186] Switching nodes in a bus switching network are also configured with matching rules, which may include matching fields and matching feature values for matching messages carrying detection tags. The matching fields may be indicated by a mask (also called a match mask). This is illustrated below with an example. As shown in FIG12 , in this example, for a single-bit trace_tag, the matching mask can be used to determine that the matching field is bit 17 in the second row of the message, and the message characteristics can be used to determine that the matching feature value is 1. Based on this, a successful match is indicated when the corresponding bit value of the message to be matched is 1. For a multi-bit trace_tag, the matching mask can be used to determine that the matching field is bits 16 to 19 in the second row of the message. The message characteristics can be used to determine that bit 16 is used to identify whether the message is a message to be counted, and the feature value corresponding to this bit can be 1. Bits 17 to 19 are used to identify the message type, and different values can correspond to different message types. The feature values corresponding to bits 17 to 19 can be 010. Based on this, when the values of bits 16 to 19 in the second row of the message to be matched are 1010, it indicates a successful match, that is, the message is a message of the specific type to be counted.
[0187] As shown in Figure 13, assuming that after the memory detection phase, detection controller 302 discovers packet loss from computing node 1.4 to computing node 1.8, it then checks all nodes on all possible paths. Assuming that the statistics of packets with trace_tag = 1 sent and received on interfaces a, b, and c at switch node 3.3 are a: (8, 8), b: (8, 6), and c: (8, 8), where x in the statistics (x, y) represents the number of packets received by the interface and y represents the number of packets sent by the interface, the sum of the input count of packets with trace_tag = 1 is 24, and the sum of the output count of packets with trace_tag = 1 is 22. Detection controller 302 can determine that packet loss occurred at switch node 3.3. In other words, the fault location may include switch node 3.3.
[0188] Based on the fault location method of the aforementioned embodiment, this application further provides a detection controller 302. This detection controller 302 is used to execute the fault location method of the aforementioned embodiment to locate faults occurring in a bus switching network within a computing cluster. The following describes the detection controller 302 of this embodiment from the perspective of functional modularity.
[0189] Referring to FIG. 14 , a schematic diagram of a detection controller 302 is shown. The detection controller 302 includes:
[0190] a communication module 3022 configured to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster including a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node being any computing node among the plurality of computing nodes or any switching node in the bus switching network;
[0191] The communication module 3022 is further configured to obtain a number of messages carrying a detection tag received by and a number of messages carrying a detection tag sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates a detection from the second node, and the messages carrying the detection tag are sent by the first node according to the detection configuration;
[0192] The fault location module 3024 is configured to determine a fault location in the bus switching network according to the number of messages carrying the detection mark received and the number of messages carrying the detection mark sent by the at least one switching node.
[0193] Exemplarily, the communication module 3022 and the fault location module 3024 may be implemented by hardware or software.
[0194] When implemented via software, the communication module 3022 and the fault location module 3024 can be applications running on a computer device, such as a computing engine. Furthermore, the application can be virtualized via virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can use virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts, providing users with VMs on demand. BMS services use virtual BMS resource pools on multiple physical hosts, providing users with BMSs on demand. Container services use virtual container resource pools on multiple physical hosts, providing users with containers on demand. A VM is a simulated virtual computer, or logically a single computer. BMS is a scalable, high-performance computing service with computing performance comparable to traditional physical machines and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.
[0195] When implemented in hardware, the communication module 3022 may be a transceiver, and the fault localization module 3024 may include at least one computing device, such as a server. Alternatively, the fault localization module 3024 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0196] In some possible implementations, the fault location module 3024 is specifically configured to:
[0197] When the number of messages carrying the detection tag received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection tag sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.
[0198] In some possible implementations, the bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically configured to:
[0199] When the number of packets carrying the detection tag sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection tag received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.
[0200] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection tag, and the message carrying the detection tag includes a memory detection message and a service message carrying the detection tag.
[0201] In some possible implementations, the fault location module 3024 is specifically configured to:
[0202] The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received and the number of messages carrying the detection mark sent by the at least one switching node.
[0203] In some possible implementations, the detection configuration further includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.
[0204] In some possible implementations, the number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.
[0205] The above describes the detection controller 302 from the perspective of functional modularization. Next, the detection controller 302 is described from the perspective of hardware implementation.
[0206] The present application also provides a detection controller 302. As shown in Figure 15, the detection controller 302 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate with each other via the bus 1502. It should be understood that the present application does not limit the number of processors and memories in the detection controller 302.
[0207] Bus 1502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG15 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 1502 may include a path for transmitting information between various components of detection controller 302 (e.g., memory 1506, processor 1504, and communication interface 1508).
[0208] The processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0209] Memory 1506 may include volatile memory, such as random access memory (RAM). Memory 1506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD). Executable program code is stored in memory 1506, and processor 1504 executes the executable program code to implement the aforementioned fault location method. Specifically, memory 1506 stores instructions for detection controller 302 to execute the fault location method.
[0210] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the detection controller 302 and other devices or a communication network.
[0211] The present application also provides a computing cluster. As shown in FIG3 , the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes.
[0212] A first node in the computing cluster is configured to receive a detection configuration sent by the detection controller 302, where the detection configuration includes information about a second node detected by the first node, where the first node and the second node are any computing node in the plurality of computing nodes or any switching node in the bus switching network;
[0213] The first node is further configured to send a message carrying a detection flag to the second node according to the detection configuration;
[0214] At least one switching node is configured to send to a detection controller the number of messages carrying a detection tag received by the at least one switching node and the number of messages carrying a detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in a bus switching network through which a first node initiates a detection from a second node.
[0215] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by the detection controller 302 or the computing cluster, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the detection controller 302 or the computing cluster to execute the fault location method.
[0216] The present application also provides a computer program product containing instructions. This computer program product can be software or a program product containing instructions that can be run on the detection controller 302, on a computing cluster, or stored on any available medium. When the computer program product is run on the detection controller 302 or the computing cluster, it causes the detection controller 302 or the computing cluster to perform the fault location method described above.
[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A fault location method, characterized in that: The method comprises: The detection controller sends a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, and the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network; The detection controller obtains the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the messages carrying the detection mark are sent by the first node according to the detection configuration; The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node.
2. The method according to claim 1, characterized in that The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: When the number of messages carrying the detection tag received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection tag sent by the first switching node, the detection controller determines that the fault location in the bus switching network includes the first switching node.
3. The method according to claim 1, characterized in that The bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the detection controller determines a fault location in the bus switching network according to the number of messages carrying a detection tag received by the at least one switching node and the number of messages carrying a detection tag sent by the at least one switching node, including: When the number of packets carrying the detection tag sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection tag received by the second switching node through the incoming interface, the detection controller determines that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.
4. The method according to any one of claims 1 to 3, characterized in that: The first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.
5. The method according to any one of claims 1 to 4, characterized in that: The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: The detection controller receives the fault record reported by the first node or the second node, and determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.
6. The method according to claim 5, characterized in that The detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.
7. The method according to any one of claims 1 to 6, characterized in that: The number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.
8. The method according to any one of claims 1 to 7, characterized in that: The detection controller is independent of the computing cluster.
9. The method according to any one of claims 1 to 7, characterized in that: The detection controller is a node in the computing cluster, or the detection controller is deployed on a node in the computing cluster.
10. A fault location method, characterized in that: Applied to a computing cluster, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the method includes: A first node in the computing cluster receives a detection configuration sent by a detection controller, wherein the detection configuration includes information of a second node detected by the first node, wherein the first node and the second node are any computing node in the plurality of computing nodes or any switching node in the bus switching network; The first node sends a message carrying a detection mark to the second node according to the detection configuration; At least one switching node sends to the detection controller the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.
11. The method according to claim 10, characterized in that The first node includes a detection agent device, and the switching node includes a statistical agent device; The first node sends a message carrying a detection mark to the second node according to the detection configuration, including: The detection agent device in the first node sends a message carrying a detection mark to the second node according to the detection configuration; The at least one switching node sends to the detection controller the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: The statistical agent in the at least one switching node sends the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node to the detection controller.
12. The method according to claim 10 or 11, characterized in that: The first node is any computing node among the plurality of computing nodes, and the detection configuration further includes a detection mode; The method further comprises: When the detection mode is global detection, the first node carries the detection mark in the service message; The message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.
13. The method according to any one of claims 10 to 12, characterized in that: The detection configuration also includes detecting expected results, and the method further includes: When the first node or the second node detects that the expected detection result is inconsistent with the actual detection result, a fault record is reported to the detection controller, and the fault record is used to trigger the detection controller to locate the fault.
14. The method according to any one of claims 10 to 13, characterized in that The number of paths from the first node to the second node is M, where M is a positive integer. The first node sends a message carrying a detection mark to the second node according to the detection configuration, including: The first node sends N messages carrying detection tags to the second node according to the detection configuration, where N is greater than or equal to M.
15. A detection controller, characterized in that: The detection controller comprises: A communication module, configured to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network; The communication module is further used to obtain the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection mark is sent by the first node according to the detection configuration; A fault location module is used to determine the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.
16. The detection controller according to claim 15, characterized in that: The fault location module is specifically used for: When the number of messages carrying the detection mark received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.
17. The detection controller according to claim 15, characterized in that: The bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically used for: When the number of packets carrying the detection mark sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection mark received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.
18. The detection controller according to any one of claims 15 to 17, characterized in that: The first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.
19. The detection controller according to any one of claims 15 to 18, characterized in that: The fault location module is specifically used for: The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.
20. The detection controller according to claim 19, characterized in that: The detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.
21. The detection controller according to any one of claims 15 to 20, characterized in that: The number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.
22. A detection controller, characterized in that: The detection controller comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the detection controller executes the method according to any one of claims 1 to 9.
23. A computing cluster, characterized in that: The computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes; A first node in the computing cluster is used to receive a detection configuration sent by a detection controller, wherein the detection configuration includes information of a second node detected by the first node, and the first node and the second node are any computing node in the multiple computing nodes or any switching node in the bus switching network; The first node is further configured to send a message carrying a detection mark to the second node according to the detection configuration; At least one switching node is used to send to the detection controller the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.
24. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9 or 10 to 14.
25. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9 or 10 to 14.
Citation Information
Patent Citations
Network fault detection method, apparatus and system
CN101360018A
Method, device and system for positioning link fault
CN102332998A
Network fault delimiting method and device and computer storage medium
CN109728931A
Network fault detection method and device, computer readable medium and electronic equipment
CN111181760A
Link fault isolation using latencies
US20200145313A1
