Fault positioning method and related equipment

By sending and receiving packets carrying detection marks in the bus switched network and analyzing packet loss, the problem of increasing difficulty in fault location in the computing cluster is solved, fast and accurate fault location is achieved, and the continuous serviceable time of the cluster is improved .

CN119071144APending Publication Date: 2024-12-03HUAWEI TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310961572.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2023-07-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

As the scale of computing clusters grows, the difficulty of fault location in bus switching networks increases, and existing tools are difficult to effectively locate fault locations, affecting the continuous serviceable time of the cluster.

Method used

A fault location method is provided, which transmits and receives packets carrying detection marks in a bus switching network through a detection controller, analyzes packet loss situations, and determines the fault locations, which is suitable for the forwarding path uncertainty under the packet-by-packet load sharing method.

Benefits of technology

It realizes rapid and accurate fault location, improves the continuous service time of the cluster, meets business needs, and avoids resource waste and resource occupation when there is no failure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071144A_ABST
    Figure CN119071144A_ABST
Patent Text Reader

Abstract

The invention provides a fault positioning method. The method is executed by a detection controller and comprises the steps that detection configuration is sent to a first node in a computing cluster, the detection configuration comprises information of a second node detected by the first node, and the computing cluster comprises a plurality of computing nodes and a bus switching network used for achieving memory access among the computing nodes, the number of messages carrying the detection marks received and sent by at least one switching node on a path from the first node to the second node in the bus network is obtained, and the messages carrying the detection marks are sent by the first node according to the detection configuration; and then determining a fault position in the bus switching network according to the number of the messages which are received and sent by the at least one switching node and carry the detection marks. Even if the bus switching network uses a packet-by-packet load sharing mode, the forwarding path is uncertain, unpredictable and non-traceable, the packet outlet position can be positioned, and then efficient fault positioning is realized.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on May 31, 2023, with application number 202310639272.5 and invention name “A method for locating faults in a bus data exchange system”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a fault location method, a detection controller, a computing cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the rise of high performance computing (HPC) or artificial intelligence (AI) technology, the demand for computing power continues to grow. As chips gradually reach the limit of Moore's Law, the computing power (i.e., computing power) of a single chip is difficult to continue to scale up vertically, and the improvement of computing power usually requires a horizontal expansion (scale out) route. For example, computing nodes including central processing units (CPU) / neural processing units (NPU) can be connected through a bus switching network to form a computing cluster.

[0004] The computing nodes in the computing cluster can achieve memory mutual access between computing nodes through the bus switching network. In other words, memory access has evolved from intra-node access to cross-node access. The bus switching network includes switching nodes, and each switching node may include at least one switching chip. When the CPU of a computing node (such as computing node 1) accesses the memory data of the NPU of another computing node (such as computing node 2), the CPU of computing node 1 can send a data access command, and the switching chip connected to the computing node 1 can convert the data access command into a data request message, and route the data request message to the switching chip connected to the NPU of the computing node 2. The switching chip connected to the NPU of computing node 2 restores the data request message to a data access command and transmits it to the NPU of computing node 2, thereby completing a cross-node memory access.

[0005] As the scale of computing clusters (such as the chip scale and bus scale of servers) continues to grow, the number of switching chips and chip interfaces that a cross-node memory access passes through increases dramatically, and the possibility of bus switching network transmission failures is increasing. When a bus switching network fails, how to locate the fault as quickly as possible to improve the continuous service time of the cluster becomes an urgent problem to be solved. Summary of the invention

[0006] The present application provides a fault location method, which can quickly locate the fault location based on the number of messages carrying detection tags received by the switching node and the number of messages carrying detection tags sent by the switching node when a bus switching network fails, thereby improving the continuous service time of the cluster. The present application also provides a detection controller, a computing cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a fault location method. The method can be executed by a detection controller. The detection controller can be software, which can be deployed on a computing node, a switching node in a computing cluster, or on an independent node. The computing node, the switching node, or the independent node executes the program code of the software to execute the fault location method of the present application. The detection controller can also be hardware, for example, the detection controller can be a server with a fault location function, and the server executes the fault location method of the present application when it is running.

[0008] Specifically, the detection controller sends a detection configuration to the first node in the computing cluster, wherein the detection configuration includes information about the second node detected by the first node, the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes, and the first node and the second node can be any computing node among the multiple computing nodes or any switching node in the bus switching network. Then, the detection controller obtains the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection mark is sent by the first node according to the detection configuration. Then, the detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node.

[0009] The method is based on the number of messages carrying detection marks received by at least one switching node in the bus switching network and the number of messages carrying detection marks sent, and analyzes the packet loss of messages carrying detection marks in the switching node to achieve fault location of the bus switching network. Even if the bus switching network uses a packet-by-packet load sharing method, resulting in an uncertain, unpredictable, and untraceable forwarding path, the packet loss position can be located, thereby achieving fault location, meeting business needs.

[0010] In some possible implementations, the detection controller may perform fault location based on the message receiving and sending conditions at the node granularity. Specifically, when the number of messages carrying the detection mark received by the first switching node in at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, the detection controller may determine that the fault location in the bus switching network includes the first switching node.

[0011] The method determines the packet loss situation by analyzing the number of messages carrying detection tags sent and received by the same switching node. According to the packet loss situation, it can be determined whether the switching node has a fault, thereby quickly determining the fault location of the node type fault and achieving efficient fault location.

[0012] In some possible implementations, the detection controller can also detect the upstream and downstream message reception and transmission conditions to locate the fault. Specifically, the bus switching network includes a first switching node and a second switching node, and the output interface of the first switching node is connected to the input interface of the second switching node. Based on this, the first switching node can be regarded as an upstream device of the second switching node, and the second switching node can be regarded as a downstream device of the first switching node. When the number of messages carrying the detection mark sent by the first switching node through the output interface (the number of messages sent upstream) is not equal to the number of messages carrying the detection mark received by the second switching node through the input interface, the detection controller determines that the fault location in the bus switching network includes a link from the output interface of the first switching node to the input interface of the second switching node.

[0013] The method determines the packet loss situation by analyzing the number of messages carrying detection tags sent by the upstream node through the outgoing interface and the number of messages carrying detection tags received by the downstream node through the incoming interface in the switching nodes connected to the interfaces. According to the packet loss situation, it can be determined whether the link between the switching nodes fails, thereby quickly determining the fault location of the link type fault and achieving efficient fault location.

[0014] In some possible implementations, the first node may be any computing node among the plurality of computing nodes, and the detection configuration further includes a detection mode. When the detection mode is global detection, the service message of the first node carries a detection tag, and the message carrying the detection tag includes a memory detection message and a service message carrying the detection tag.

[0015] In this way, fault detection can be performed while the business is running without waiting for the business to be suspended. In addition, the business message carrying the detection tag can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and realize resource reuse.

[0016] In some possible implementations, the detection controller may trigger fault location when the detection task detects a fault. Specifically, the detection controller receives a fault record reported by the first node or the second node, and determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node.

[0017] The method performs fault location according to the number of messages carrying detection marks received by the switching node and the number of messages carrying detection marks sent by the switching node when a fault is detected, thereby avoiding resource waste caused by fault location when no fault occurs and improving resource utilization.

[0018] In some possible implementations, the detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result and the actual detection result are inconsistent. If the detection access mode is a read mode, the expected detection result can be the data requested to be read, and the data can be pre-agreed data. If the detection access mode is a write mode, the expected detection result can be the detection request content, such as data to be written.

[0019] In this method, the first node or the second node can realize fault detection by comparing the consistency between the expected detection result and the actual detection result, and when a fault is detected, report the fault record to the detection controller so that the detection controller triggers fault location, thereby improving the fault location efficiency.

[0020] In some possible implementations, the number of messages carrying a detection tag sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node. For example, the number of messages carrying a detection tag sent by the first node to any second node is greater than or equal to the number of load sharing paths from the first node to the second node. In this way, the paths from the first node to the second node can be covered as much as possible, the comprehensiveness of the fault location analysis can be improved, and the accuracy of the fault location can be improved.

[0021] In some possible implementations, the detection controller is independent of the computing cluster, so that the detection controller can avoid occupying the service resources in the computing cluster when locating the fault, thus ensuring the normal operation of the service.

[0022] In some possible implementations, the detection controller may be a node in a computing cluster, or the detection controller may be deployed on a node in a computing cluster, so that existing nodes can be reused, system complexity can be reduced, and additional network bandwidth overhead can be reduced.

[0023] In a second aspect, the present application provides a fault location method, which is applied to a computing cluster, wherein the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes.

[0024] Specifically, a first node in a computing cluster receives a detection configuration sent by a detection controller, where the detection configuration includes information about a second node detected by the first node, the first node and the second node are any computing node among the multiple computing nodes or any switching node in the bus switching network, and then the first node sends a message carrying a detection tag to the second node according to the detection configuration, and at least one switching node sends to the detection controller the number of messages carrying the detection tag received by at least one switching node and the number of messages carrying the detection tag sent by at least one switching node, wherein at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.

[0025] The method is based on the number of messages carrying detection marks received by at least one switching node in the bus switching network and the number of messages carrying detection marks sent, and analyzes the packet loss of messages carrying detection marks in the switching node to achieve fault location of the bus switching network. Even if the bus switching network uses a packet-by-packet load sharing method, resulting in an uncertain, unpredictable, and untraceable forwarding path, the packet loss position can be located, thereby achieving fault location, meeting business needs.

[0026] In some possible implementations, the first node includes a detection agent, and the switching node includes a statistical agent. Accordingly, the detection agent in the first node sends a message carrying a detection tag to the second node according to the detection configuration; the statistical agent in the at least one switching node sends the number of messages carrying the detection tag received and the number of messages carrying the detection tag sent by the at least one switching node to the detection controller.

[0027] Among them, the detection agent device, the statistical agent device and the detection controller can cooperate to achieve fault location.

[0028] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration further includes a detection mode. When the detection mode is global detection, the first node carries the detection tag in a service message. The message carrying the detection tag includes a memory detection message and a service message carrying the detection tag.

[0029] In this way, fault detection can be performed while the business is running without waiting for the business to be suspended. In addition, the business message carrying the detection tag can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and realize resource reuse.

[0030] In some possible implementations, the detection configuration further includes an expected detection result. When the first node or the second node detects that the expected detection result is inconsistent with the actual detection result, a fault record is reported to the detection controller. The fault record is used to trigger the detection controller to locate the fault.

[0031] The method performs fault location when a fault is detected, thereby avoiding resource waste caused by performing fault location when no fault occurs, and improving resource utilization.

[0032] In some possible implementations, the number of paths from the first node to the second node is M, where M is a positive integer, and accordingly, the first node may send N messages carrying detection tags to the second node according to the detection configuration, where N is greater than or equal to M. In this way, the path from the first node to the second node may be covered as much as possible, thereby improving the comprehensiveness of fault location analysis and thereby improving the accuracy of fault location.

[0033] In a third aspect, the present application provides a detection controller. The detection controller comprises:

[0034] A communication module, configured to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network;

[0035] The communication module is further used to obtain the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection mark is sent by the first node according to the detection configuration;

[0036] A fault location module is used to determine the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0037] In some possible implementations, the fault location module is specifically used to:

[0038] When the number of messages carrying the detection mark received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.

[0039] In some possible implementations, the bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically configured to:

[0040] When the number of packets carrying the detection mark sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection mark received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.

[0041] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.

[0042] In some possible implementations, the fault location module is specifically used to:

[0043] The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0044] In some possible implementations, the detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.

[0045] In some possible implementations, the number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.

[0046] In a fourth aspect, the present application provides a detection controller. The detection controller includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the detection controller executes the method described in any implementation of the first aspect of the present application.

[0047] In a fifth aspect, the present application provides a computing cluster. The computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes;

[0048] A first node in the computing cluster is used to receive a detection configuration sent by a detection controller, wherein the detection configuration includes information about a second node detected by the first node, and the first node and the second node are any computing node in the multiple computing nodes or any switching node in the bus switching network;

[0049] The first node is further configured to send a message carrying a detection mark to the second node according to the detection configuration;

[0050] At least one switching node is used to send to the detection controller the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.

[0051] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a detection controller to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.

[0052] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing cluster to execute the fault location method described in the second aspect or any one of the implementations of the second aspect.

[0053] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed on a detection controller, enables the detection controller to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.

[0054] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing cluster, enables the computing cluster to execute the fault location method described in the first aspect or any one of the implementations of the first aspect.

[0055] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0057] Figure 1 A schematic diagram of the architecture of a computing cluster provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a process of transmitting a memory access request by a switching chip in a bus switching network node provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the architecture of a fault location system provided in an embodiment of the present application;

[0060] Figure 4A A flowchart of a fault location method provided in an embodiment of the present application;

[0061] Figure 4B A flowchart of another fault location method provided in an embodiment of the present application;

[0062] Figure 5 An interactive flow chart of a fault location method provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of counting messages carrying detection tags provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of accessing memory by reading provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of accessing memory by writing provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of an exception handling process provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of the life cycle of a detection task provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of an application scenario of a fault location method provided in an embodiment of the present application;

[0069] Fig.12 A schematic diagram of a matching rule provided in an embodiment of the present application;

[0070] Fig.13 A schematic diagram of determining a fault location provided in an embodiment of the present application;

[0071] Fig.14 A schematic diagram of the structure of a detection controller provided in an embodiment of the present application;

[0072] Fig.15A hardware structure diagram of a detection controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0074] To facilitate understanding, some technical terms involved in this application are first introduced.

[0075] A bus switching network, also referred to as a bus network, is a network that implements data exchange through a bus. A bus switching network can generally include at least one switching node, which can be, for example, a bus switch. A bus switching network can adopt different network architectures, including but not limited to a Clos network architecture and a dragonfly network architecture. For ease of description, the following example uses a bus switching network using a Clos network architecture.

[0076] like Figure 1 As shown in Figure 1, a simple Clos network consists of the following switching layers: leaf switching layer and spine switching layer. The leaf switching layer includes access switches (also called leaf switches) that aggregate traffic from computing nodes (such as servers) and are directly connected to the spine switching layer. The spine switching layer includes spine switches that are directly connected to leaf switches.

[0077] The leaf switching layer is located at the bottom layer of the bus switching network, so the leaf switches in the leaf switching layer are also called level 1 (L1) switches and L1 switching nodes. The spine switching layer is located at the upper layer of the bus switching network, so the spine switches in the spine switching layer are also called level 2 (L2) switches and L2 switching nodes. Figure 1 In the example of FIG. 1 , the bus switching network may include leaf switches L1 - 1 to L1 - 7 and spine switches L2 - 1 to L2 - 8 .

[0078] The bus switching network can connect different computing nodes to form a computing cluster (also called a cluster computing system or super node), thereby increasing computing power through horizontal expansion. Figure 1As an example, a computing node includes a processor, which may include but is not limited to a central processing unit (CPU) and a neural processing unit (NPU). The CPU / NPU in the computing node can implement memory access between computing nodes (cross-node memory access) through a bus switching network.

[0079] The following describes how to implement memory access between computing nodes by switching nodes.

[0080] A switching node may include one or more switching chips that support data exchange via a bus. Figure 2 As shown, a switching node may include the following switching chips: switching chip-1, switching chip-2, and switching chip-3. When CPU 201 accesses the memory data at the location 0x3FE0 of NPU 202 (i.e., the memory address is 0x3FE0), CPU 201 sends a data request command to the adjacent switching chip-1. Switching chip-1 queries the address table according to the memory address 0x3FE0, obtains the network address (network identifier, NID) 127 corresponding to the memory address, converts the data request command into a data request message, and sets the network address NID to 127 in the data request message, then queries the routing table to obtain the output interface 16, and then forwards it to switching chip-2. After the data request message is forwarded to switching chip-3, switching chip-3 restores the data request message to a data request command and transmits it to NPU 202, thereby completing the transmission process of a memory access request.

[0081] As the scale of computing clusters (such as the chip scale and bus scale of servers) continues to grow, the number of switching chips and chip interfaces that a cross-node memory access passes through increases dramatically, and the possibility of bus switching network transmission failures is increasing. When a bus switching network fails, how to locate the fault location as quickly as possible to improve the continuous service time of the cluster becomes an urgent problem to be solved. Fault location tools widely used in the industry include traceroute, ping, or pingmesh. However, most of the above tools are built for Internet Protocol (IP) services and use media access control (MAC) addresses or IP addresses for addressing. Bus switching networks usually use memory addresses for addressing when performing memory access. Fault location tools such as traceroute, ping, or pingmesh are difficult to meet the fault location requirements when performing memory access through bus switching networks.

[0082] In view of this, the present application provides a fault location method. The method can be performed by a detection controller. The detection controller is also called a super node controller, which is used to locate the fault when the bus switching network in the super node (i.e., computing cluster) fails. Among them, the detection controller can be software, and the software can be deployed in the computing node, switching node, or independent node in the computing cluster. The computing node, switching node or independent node executes the program code of the software to execute the fault location method of the present application. The detection controller can also be hardware, for example, the detection controller can be a server with a fault location function, and the server executes the fault location method of the present application when it is running.

[0083] Specifically, the detection controller can send a detection configuration to the first node in the computing cluster, and the detection configuration includes information about the second node detected by the first node, wherein the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between multiple computing nodes, and the first node and the second node can be any computing node among the multiple computing nodes or any switching node in the bus switching network. Then, the detection controller can obtain the number of messages carrying a detection tag (trace_tag) received by at least one switching node and the number of messages carrying the detection tag sent. Among them, at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection tag is sent by the first node according to the detection configuration. Then, the detection controller determines the fault location according to the number of messages carrying the detection tag received by at least one switching node and the number of messages carrying the detection tag sent.

[0084] The method determines the packet loss of the message carrying the detection mark in the switching node based on the number of messages carrying the detection mark received by at least one switching node in the bus switching network and the number of messages carrying the detection mark sent, thereby realizing the fault location of the bus switching network. Even if the bus switching network uses a packet-by-packet load sharing method, which leads to an uncertain, unpredictable, and non-traceable forwarding path, the packet loss position can be located, thereby realizing the fault location, thus meeting the business needs.

[0085] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the fault location system of the embodiment of the present application is first introduced below with reference to the accompanying drawings.

[0086] See also Figure 3The schematic diagram of the architecture of a fault location system is shown, the fault location system 300 includes a detection controller 302. Further, the fault location system 300 may also include a detection agent 304 and a statistical agent 306. The detection agent 304 may be deployed on a computing node or a switching node in a computing cluster, and the statistical agent 306 may be deployed on a switching node. Similar to the detection controller 302, the detection agent 304 and the statistical agent 306 may be software, and the node (such as a computing node or a switching node) on which the software is deployed executes the program code of the software, thereby executing the steps performed by the detection agent 304 and the statistical agent 306 in the fault location method.

[0087] Specifically, the detection controller 302 is used to send a detection configuration to the first node in the computing cluster, and the detection configuration includes information about the second node that the first node initiates detection. Among them, the first node and the second node can be any computing node among multiple computing nodes or any switching node in the bus switching network. In some embodiments, the detection controller 302 can send detection configurations to all nodes in the computing cluster (including the first node and the second node), and nodes of different role types (such as computing nodes and switching nodes) use different information in the detection configuration. For ease of description, this application takes the first node as an example of a computing node.

[0088] Among them, the detection controller 302 can send a detection task to the first node, and the detection task includes a detection configuration. In some examples, the detection controller 202 can also send a detection configuration to the first node, and the first node generates a detection task according to the detection configuration. The detection task can be performed when the user service is suspended. In other words, the execution timing of the detection task can be when the user service is suspended. In this case, the messages sent from the first node are all memory detection messages. The detection task can also be performed when the user service is running. In other words, the execution timing of the detection task can be when the user service is running, the detection mode is global detection, and the messages sent from the first node include memory detection messages and service messages carrying detection tags.

[0089] The detection agent 304 deployed in the first node is used to send a message carrying a detection tag according to the detection configuration. Among them, the message carrying the detection tag is used to detect connectivity by accessing the memory across nodes. The message carrying the detection tag sent by the detection agent 304 can be a request message. The source node of the request message is the first node, and the destination node of the request message is the second node. The detection agent 304 on the second node is allocated a memory space (detection memory) for access by the detection agent 304 of other nodes (such as the first node) to detect connectivity. The second node receives the request message and can detect connectivity and return a response message corresponding to the request message. It should be noted that the response message corresponding to the request message can also carry a detection tag. The detection tag can be set to a target value, for example, trace_tag can be set to 1, so that the switching node can recognize the detection tag and then perform message statistics. A message carrying a detection tag can cover a load sharing path and a routing table entry. Multiple messages carrying detection tags generated according to the detection configuration can traverse all load sharing paths and all routing table entries during the forwarding process of the bus switching network, so as to comprehensively detect problems in the bus switching network.

[0090] The statistical agent device 306 deployed in at least one switching node is used to count the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node. Wherein, when the response message carries the detection mark, the message carrying the detection mark includes the request message and the response message corresponding to the request message; when the response message does not carry the detection mark, the message carrying the detection mark includes the request message.

[0091] In some possible implementations, when the detection mode is global detection, the service message of the computing node that enables the detection function also carries the detection mark. For example, when the first node is a computing node, the service message of the first node also carries the detection mark. Correspondingly, the message carrying the detection mark includes a memory detection message (such as a memory detection request message, or a memory detection request message and a memory detection response message) and a service message carrying the detection mark. In other words, when the statistical agent 306 counts the number of messages carrying the detection mark received by the switching node and the number of messages carrying the detection mark sent, it can count the number of memory detection messages and service messages carrying the detection mark received by the switching node and the number of memory detection messages and service messages carrying the detection mark sent. Among them, the statistical agent 306 can count the number of messages carrying the detection mark received at the input interface of the port of the switching node, and count the number of messages carrying the detection mark sent at the output interface of the port of the switching node.

[0092] The detection controller 302 is further configured to obtain the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, and determine the fault location according to the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node. The at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.

[0093] It should be noted that the statistical agent device 306 can periodically report statistical data to the detection controller 302, and the statistical data includes the number of messages carrying the detection mark received by the switching node and the number of messages carrying the detection mark sent. The reporting method can be active reporting or passive response to the statistical collection request of the detection controller 302. The detection controller 302 obtains the statistical data reported by the statistical agent device 306, thereby obtaining the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent.

[0094] In some possible implementations, the detection controller 302 may also receive a fault record reported by the first node or the second node, where the fault record is determined by the first node or the second node based on the expected detection result and the actual detection result. Accordingly, the detection controller 302 may trigger a fault location process to determine the fault location when receiving the fault record. For example, the detection controller 302 may determine the fault location in the bus switching network based on the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node after receiving the fault record reported by the first node or the second node.

[0095] It should be noted that the detection controller 302 can determine the fault location in combination with the fault record. For example, the detection controller 302 can obtain the source node and the destination node from the fault record, obtain the number of messages carrying the detection mark received by the switching node and the number of messages carrying the detection mark sent by the switching node on the path from the source node to the destination node (for example, a load sharing path), and determine the fault location based on the number, so that the detection range can be reduced and the detection efficiency can be improved.

[0096] Figure 3 The example of the detection controller 302 receiving the fault record reported by the destination node does not constitute a limitation on the technical solution of the present application. Figure 3 The control flow (e.g., the control flow for sending detection configuration, collecting statistical data or fault records) and the detection flow (e.g., the data flow of the message carrying the detection mark) are also shown through different line types. The above control flow can be transmitted through the control plane, and the data flow can be transmitted through the data plane.

[0097] Next, the fault location method of the embodiment of the present application will be introduced with reference to the accompanying drawings.

[0098] See also Figure 4A A flow chart of a fault location method is shown, the method is applied to the detection controller 302, and the method includes:

[0099] S402: The detection controller 302 sends a detection configuration to a first node in the computing cluster.

[0100] The computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes. The detection configuration includes information about the second node that the first node initiates the detection. The first node and the second node can be any computing node among the multiple computing nodes or any switching node in the bus switching network. The first node can include one or more nodes, and the second node can include one or more nodes. The path from the first node to the second node includes a forwarding path of the bus switching network. For example, the path from the first node to the second node can cover all forwarding paths of the bus switching network, so that all forwarding paths can be traversed subsequently to locate the fault.

[0101] In some possible implementations, the computing cluster includes multiple node groups, and different computing nodes in the same node group are directly connected to the same switching node. Accordingly, the second node may include computing nodes at the same position in the remaining node groups other than the node group to which the first node belongs. In this way, the path from the first node to the second node may cover the forwarding path of the bus switching network.

[0102] Specifically, when the detection controller 302 sends the detection configuration to the first node, it can send a detection task, and the detection task includes the above-mentioned detection configuration. Alternatively, the detection controller 302 can directly send the detection configuration so that the first node generates a detection task according to the detection configuration. The information of the second node included in the detection configuration can be represented by a node list. In some examples, the node list can record the identifier of the second node. This embodiment uses a memory access method to locate the fault. Based on this, the detection configuration can also include an expected detection result. Based on the expected detection result and the actual detection result, it can be determined whether a fault occurs. If so, fault location can be triggered.

[0103] Further, the detection configuration may also include a detection access mode. When the detection access mode is read access, the first node (source node) may determine whether the bus switching network fails according to the expected detection result and the actual detection result. When the detection access mode is write access, the second node (destination node) may determine whether the bus switching network fails according to the expected detection result and the actual detection result. Among them, the detection configuration may also include a detection access memory address, so as to read data from the corresponding memory address or write data to the corresponding memory address, thereby realizing the detection connectivity. Considering that the detection task may be a periodic task, the detection configuration may also include at least one of the detection frequency, the detection time interval, the number of detections, the detection start time, and the detection end time. In order to facilitate the fine-grained statistics of the number of messages, the detection configuration may also indicate the path number. The path number may be the number of one or more paths. Among them, multiple paths may be equivalent paths, such as multiple paths with the same source node and the same destination node. In this way, when counting messages, messages with different path numbers may also be counted separately.

[0104] S404: The detection controller 302 obtains the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node.

[0105] The message carrying the detection mark is used to detect connectivity by accessing the memory across nodes. The message carrying the detection mark includes a request message. The source node of the request message is the first node, and the destination node of the request message is the second node. In some possible implementations, the response message corresponding to the request message carrying the detection mark may also carry the detection mark. The source node of the response message is the second node, and the destination node of the response message is the first node.

[0106] The message carrying the detection mark (for example, the request message carrying the detection mark) can be generated by the first node according to the detection configuration. Specifically, the first node receives the detection task including the above detection configuration, and can generate a request message carrying the detection mark in response to the detection task. Or the first node receives the detection configuration, can generate a detection task, and then generate a request message carrying the detection mark in response to the detection task. Then the first node can send the request message carrying the detection mark to the second node. Among them, at least one switching node in the bus switching network can forward the request message to the second node. At least one switching node can be a switching node in the bus switching network through which the first node initiates the detection to the second node.

[0107] In order to ensure that the message carrying the detection mark (such as the request message) can cover as many load sharing paths as possible, the first node can send multiple messages carrying the detection mark to the second node (for example, each node in the second node). In some examples, for any node in the second node, the number of messages carrying the detection mark (such as the request message) sent by the first node to the second node is greater than or equal to the number of paths (such as the load sharing paths) from the first node to the second node. Assuming that there are M load sharing paths from the first node to the second node, at least N detection requests need to be sent for each detection, where N>=M. In addition, in order to ensure that data can be obtained from other nodes each time it is accessed, the detection memory of the second node can usually be configured as non-cacheable to avoid hitting data in the cache and affecting the detection results.

[0108] The second node receives the request message and can identify the detection tag in the request message. When the request message includes the detection tag, the detection memory of the second node can be accessed to detect connectivity. The second node can generate a response message carrying the detection tag based on the detection result and return the response message carrying the detection tag to the first node. At least one switching node in the bus switching network can forward the response message to the first node.

[0109] The number of messages carrying the detection mark received by the switching node is determined by identifying the detection mark in the received message, and the number of messages carrying the detection mark sent by the switching node is determined by identifying the detection mark in the sent message. Since the request message and the response message carrying the detection mark pass through the switching node, the statistical agent device 306 deployed in the switching node can determine the number of messages carrying the detection mark received by identifying the detection mark in the received message, and determine the number of messages carrying the detection mark sent by identifying the detection mark in the sent message. It should be noted that the above number can also be determined by other devices or equipment, and this embodiment does not limit this.

[0110] Furthermore, the packets carrying the detection mark counted by the statistical agent 306 deployed in the switching node may be different when the detection task is executed at different times.

[0111] When the detection task execution timing is when the user service is suspended, the messages sent from the first node are all memory detection messages. Among them, the memory detection message is a message dedicated to memory detection. In this case, the messages carrying the detection mark counted by the statistical agent 306 include memory detection messages.

[0112] When the detection task execution time is when the user service is running, it means that the detection mode is global detection, and the service message of the computing node that enables the detection function also carries the detection mark. Based on this, when the first node is any computing node among the multiple computing nodes, the message sent from the first node includes the memory detection message and the service message carrying the detection mark. The message carrying the detection mark includes the memory detection message and the service message carrying the detection mark. Accordingly, the message carrying the detection mark counted by the statistical agent device 306 includes the memory detection message and the above-mentioned service message carrying the detection mark.

[0113] S406. The detection controller 302 determines the fault location according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0114] The detection controller 302 can perform node granularity analysis to determine the fault location. Specifically, the detection controller 302 can check whether the number of messages carrying the detection mark received by the same switching node in at least one switching node is equal to the number of messages carrying the detection mark sent by the same switching node. When the number of messages carrying the detection mark received by the first switching node in at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, the detection controller 302 determines that the fault location in the bus switching network includes the first switching node.

[0115] The detection controller 302 can also analyze from the upstream and downstream perspectives to determine the fault location. Specifically, the bus switching network includes a first switching node and a second switching node, and the output interface of the first switching node is the input interface of the second switching node. Therefore, the first switching node can be regarded as an upstream node of the second switching node, and the second switching node can be regarded as a downstream node of the first switching node. The detection controller 302 can check whether the number of messages carrying the detection mark sent by the first output interface of the first switching node and the number of messages carrying the detection mark received by the first input interface of the second switching node are equal. When the number of messages carrying the detection mark sent by the first switching node through the output interface is not equal to the number of messages carrying the detection mark received by the second switching node through the input interface, the detection controller 302 determines that the fault location in the bus switching network includes a link from the output interface of the first switching node to the input interface of the second switching node.

[0116] Among them, the link from the outbound interface of the first switching node to the inbound interface of the second switching node may include a network transmission channel from the outbound interface of the first switching node to the inbound interface of the second switching node, or the outbound interface of the first switching node and the inbound interface of the second switching node. In other words, when a failure occurs in the link from the outbound interface of the first switching node to the inbound interface of the second switching node, it may be a failure of the outbound interface of the first switching node, a failure of the inbound interface of the second switching node, or a failure of the network transmission channel.

[0117] Based on the above description, the fault location method of the present application is based on the number of messages carrying detection tags received by at least one switching node in the bus switching network, and the number of messages carrying detection tags sent, and analyzes the packet loss of messages carrying detection tags in the switching node to achieve fault location of the bus switching network. Even if the bus switching network uses a packet-by-packet load sharing method, resulting in an uncertain, unpredictable, and non-traceable forwarding path, the packet loss position can be located, thereby achieving fault location and meeting business needs.

[0118] Figure 4A The fault location method of the present application is introduced from the perspective of the detection controller 302. Next, the fault location method of the present application is introduced from the perspective of a computing cluster.

[0119] See also Figure 4B A flowchart of a fault location method is shown, the method is applied to a computing cluster, the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes, the method includes:

[0120] S403 . The first node in the computing cluster receives the detection configuration sent by the detection controller 302 .

[0121] The detection configuration includes information of the second node detected by the first node. The first node and the second node may be any computing node among a plurality of computing nodes or any switching node in a bus switching network.

[0122] S405: The first node sends a message carrying a detection mark to the second node according to the detection configuration.

[0123] When the detection controller 302 sends down the detection configuration in the form of carrying the detection configuration in the detection task, the first node can respond to the detection task, generate a message carrying the detection mark according to the detection configuration, such as a request message carrying the detection mark, and then send the message carrying the detection mark to the second node.

[0124] When the detection controller 302 directly sends down the detection configuration, the first node can generate a detection task according to the detection configuration, and then generate a message carrying the detection tag according to the detection configuration in response to the detection task, and then send the message carrying the detection tag to the second node.

[0125] The first node includes a detection agent. The first node can send a message carrying a detection tag through the detection agent. For example, the detection agent in the first node sends a message carrying a detection tag to the second node according to the detection configuration.

[0126] In some possible implementations, the first node is any computing node among the multiple computing nodes. The detection configuration may also indicate a detection mode. When the detection mode is global detection, the first node carries a detection tag in the service message. Accordingly, the message carrying the detection tag includes a memory detection message and a service message carrying the detection tag. In this way, fault detection can be performed while the service is running without having to wait for the service to be suspended, and the service message carrying the detection tag to participate in the detection can reduce the number of additional memory detection messages sent, avoid occupying more resources due to fault detection, and realize resource reuse.

[0127] It should also be noted that, in order to make the message cover as many paths as possible, the number of messages carrying the detection mark sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node. Assuming that the number of paths (such as load sharing paths) from the first node to the second node is M, M is a positive integer, the first node can send N messages carrying the detection mark to the second node according to the detection configuration, where N is greater than or equal to M.

[0128] Furthermore, the second node can also perform memory detection according to the message carrying the detection mark to obtain the actual detection result. Wherein, when the detection access mode is read, the second node can also return the actual detection result. In this way, both the first node and the second node can obtain the actual detection result. When the first node or the second node detects that the expected detection result and the actual detection result are inconsistent, a fault record can be reported to the detection controller 302. Wherein, the fault record is used to trigger the detection controller to locate the fault.

[0129] S407 . At least one switching node in the bus switching network included in the computing cluster sends to the detection controller 302 the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0130] Among them, at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node. The switching node includes a statistical agent. Accordingly, the switching node can count the number of messages through the statistical agent and report it. In specific implementation, the statistical agent in at least one switching node sends the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent to the detection controller 302. In this way, the detection controller 302 can locate the fault based on the number of messages carrying the detection mark received by the switching node and the number of messages carrying the detection mark sent.

[0131] Based on the above description, it can be known that the method reports the number of messages carrying detection marks received by at least one switching node in the bus switching network and the number of messages carrying detection marks sent to the detection controller 302. The detection controller 302 can analyze the packet loss of the messages carrying detection marks in the switching node to achieve fault location of the bus switching network. Even if the bus switching network uses a packet-by-packet load sharing method, resulting in an uncertain, unpredictable, and non-traceable forwarding path, the packet loss position can be located, thereby achieving fault location and meeting business needs.

[0132] The detection controller 302 is the organizer of the detection task, responsible for coordinating the computing nodes (for example, the detection agent device 304 deployed in the computing node) and the switching nodes (for example, the statistical agent device 306 deployed in the switching node) to cooperate and complete the detection task. The fault location method of the present application is described below from the perspective of the interaction between the detection controller 302, the computing nodes (or the detection agent device 304 deployed in the computing nodes), and the switching nodes (the statistical agent device 306 deployed in the switching nodes).

[0133] See also Figure 5 An interactive flow chart of a fault location method is shown, the method comprising:

[0134] S502: The detection controller 302 issues a detection task.

[0135] A detection task refers to a task of generating a message carrying a detection tag and transmitting the message through a bus switching network to perform connectivity detection. The detection task can be issued by the detection controller 302, or the detection controller 302 can issue a detection configuration (e.g., a detection task description) to the first computing node, and the first computing node generates according to the detection configuration. Among them, the detection configuration may include information of a second computing node (detection request target node) that is detected by the first computing node (also referred to as a detection execution node, a detection request source node). In some examples, the detection task can cover the interface input and output directions of the switching node (e.g., all interface input and output directions), and the forwarding table items of the switching node (e.g., all forwarding table items).

[0136] In some possible implementations, the detection configuration further includes at least one of the detection frequency, the number of detections, the detection start time, the detection end time, the detection rate (i.e., the detection message sending rate), and the detection time interval. Among them, the detection frequency, the number of detections, the detection start time, the detection end time, the detection rate, and the detection time interval can indicate the detection task repetition mode, and the first computing node can repeatedly initiate the detection task according to the above parameters in the detection configuration. For example, the first computing node can periodically initiate the detection task according to the detection time interval.

[0137] Furthermore, the detection configuration may also include a detection expected result. If the detection access mode is a read mode, the detection expected result may be the data requested to be read. If the detection access mode is a write mode, the detection expected result may be the detection request content, such as the data to be written.

[0138] It should be noted that Figure 5 Taking the example of the detection controller 302 directly sending the detection task, in other possible implementations of the embodiment of the present application, the detection controller 302 may also send the detection configuration, and the first computing node generates the detection task according to the detection configuration.

[0139] S504 . The detection agent device 304 deployed in the first computing node sends a request message carrying a detection tag in response to the detection task.

[0140] The detection agent 304 can be deployed on a computing node. The detection agent 304 can be a computing task execution engine of the computing node, or it can be deployed on the computing node in the form of a virtual machine (VM), and the computing task runs in the VM. The detection agent 304 can first allocate a piece of memory for access by the detection agent 304 deployed in other nodes, and set the memory data to an agreed value. The memory space allocated by the detection agent 304 must be large enough so that all interleaved channels can be covered when the detection agent 304 of other nodes accesses it. It should be noted that if the detection access is implemented in a write manner, the detection agent 304 can allocate multiple blocks of memory for detection access by other nodes.

[0141] Specifically, the detection agent 304 deployed in the first computing node can carry the detection tag trace_tag in the message, send several request messages carrying the detection tag to other nodes according to the detection task, wait for the response message carrying the detection tag, and verify the detection result. According to different detection access modes, the request message type can be different. For example, when the detection access mode is a read mode, the request message can be a message based on load (load, LD) or a message based on direct memory access (direct memory access, DMA) READ. When the detection access mode is a write mode, the request message can be a message based on store (store, ST) or a message based on DMA WRITE. Among them, LD and ST instructions are memory instructions of the CPU. The LD instruction is used to read a single data of the memory address, which can usually be 8 to 256 bits. The ST instruction is used to write a single data (for example, 8 to 256 bits) to the memory address. DMA READ / DMA WRITE can be used for reading and writing large segments of data, such as reading and writing data of more than 64 bytes.

[0142] Furthermore, in order to ensure that the request message carrying the detection tag can cover as many load sharing paths as possible, the first computing node can send multiple request messages carrying the detection tag to the second computing node. Assuming that there are M paths (for example, load sharing paths) from the first computing node to the second computing node, the first computing node can send at least N request messages carrying the detection tag each time it detects, where N>=M. In addition, in order to ensure that data can be obtained from other nodes each time it is accessed, the detection memory of the second computing node can usually be configured as non-cacheable to avoid hitting data in the cache and affecting the detection results.

[0143] When the detection task indicates a detection start time, the detection agent 304 deployed in the first computing node can also set a timer, and when the time set by the timer arrives, send a message carrying a detection mark to request cross-node access to memory and detect connectivity. When the detection task indicates a detection end time, the detection agent 304 deployed in the first computing node can also stop sending messages carrying detection marks when the detection end time arrives. For example, when global detection is turned on, the detection agent 304 deployed in the first computing node can also stop sending memory detection messages when the detection end time arrives, and turn off the function of carrying detection marks in business messages.

[0144] S506. The statistics agent 306 deployed in the switching node triggers message statistics.

[0145] The statistical agent 306 can trigger the statistics of the number of sent messages carrying the detection mark and the number of received messages carrying the detection mark. Specifically, the statistical agent 306 can initialize the number of sent messages carrying the detection mark to 0, and the number of received messages carrying the detection mark to 0. When the outbound interface of the switching node sends a message carrying the detection mark, the number of sent messages carrying the detection mark is updated, for example, the number of sent messages carrying the detection mark is increased by 1. When the inbound interface of the switching node receives a message carrying the detection mark, the number of received messages carrying the detection mark is updated, for example, the number of received messages carrying the detection mark is increased by 1. It should be noted that the statistical agent 306 can count the number of messages passing through each interface respectively, or count the number of messages carrying the detection mark sent by the switching node and the number of messages carrying the detection mark received without distinguishing the interfaces.

[0146] The detection mark may include various forms, and the following describes the forms of different detection marks with examples.

[0147] In a first possible implementation, the detection mark may be a bit in the message used to identify whether statistics are needed. When the value of the bit is a set value (eg, 1), it indicates that the message needs to be counted.

[0148] In the second implementation, the detection tag can be a plurality of bits in the message, and the plurality of bits can have a plurality of values. Among all possible values ​​of the detection tag, a part of the values ​​are used to identify the message as a message to be counted, and the remaining values ​​are used to identify the message as a message not to be counted. For example, when the detection tag includes two bits, all possible values ​​of the detection tag are: 0 (binary representation is 00), 1 (binary representation is 01), 2 (binary representation is 10), 3 (binary representation is 11), wherein the value 0 is used to identify the message as a message not to be counted, and the values ​​1, 2, and 3 are used to identify the message as a message to be counted. Furthermore, the value for identifying the message as a message to be counted can be further subdivided, for example, the value is also used to identify the category of the message to be counted. For example, the message to be counted can be classified according to the source node, and some values ​​of the detection tag can be used to identify the message to be counted of each source node.

[0149] In a third possible implementation, the detection tag may be a plurality of bits in a message, wherein one bit is used to identify whether the message is a message to be counted, and the remaining bits are used to identify the category of the message to be counted. Still taking the example that the detection tag includes two bits, the first bit identifies whether the message is a message to be counted, and the second bit identifies the category of the message to be counted, for example, the second bit may identify the source node of the message to be counted.

[0150] The statistical agent 306 is deployed on the switching node. Based on this, the statistical agent 306 can first perform statistical matching content configuration on the switching node. The statistical matching content includes but is not limited to matching fields and matching feature values. For example, the statistical agent 306 can be configured: the matching field is the trace_tag field, and the matching feature value is 1, which is used to identify the message with trace_tag=1. Figure 6 When the message is forwarded in the switching node, the statistical agent device 306 performs feature matching. If the field value of the matching field of the message meets the matching feature value, for example, trace_tag=1 in the message, the message is determined to be a message carrying a detection tag. The statistical agent device 306 performs statistics on the message in the input interface and the output interface direction respectively.

[0151] S508: The detection agent 304 deployed in the second computing node receives the request message carrying the detection tag forwarded by the switch node, and performs memory detection.

[0152] S509: The detection agent device 304 deployed in the second computing node generates a response message, and returns the response message to the first computing node.

[0153] The detection agent device 304 deployed in the second computing node can perform memory detection in a read mode or a write mode. The following is an explanation of each of the above with reference to examples.

[0154] See also Figure 7 A schematic diagram of memory detection by reading is shown, in which the first computing node (source node) adopts a reading method, such as LD, DMA READ, to read the data in the detection memory of the second computing node (destination node), wherein the detection memory of the second computing node is a non-cacheable memory. In this example, the actual detection result can be the data read from the detection memory. When the response message carrying the above data is forwarded to the first computing node by the switching nodes 1, 2, 3 (denoted as SW1, SW2, SW3) in the bus switching network, the first computing node can obtain the data (or value) of the memory address in the detection memory of the second computing node.

[0155] See also Figure 8 The diagram shows a memory detection by writing, in which the first computing node (source node) writes data to the detection memory of the second computing node (destination node) by writing, for example, ST, DMA WRITE, wherein the detection memory of the second computing node is a non-cacheable memory. In this example, the actual detection result may be the data written to the detection memory of the second computing node. The second computing node may also detect whether the detection memory is rewritten to the expected detection result (pre-agreed data) to determine whether a fault occurs. If a block of memory is not rewritten successfully, it indicates that a fault has occurred in the bus switching network. Figure 8 As an example, when the switching node 2 (such as SW2) in the bus switching network fails, it may cause a block of memory in the detection memory of the second computing node to fail to be rewritten. The second computing node detects that a block of memory has failed to be rewritten and determines that the bus switching network has failed.

[0156] If the request message carrying the detection mark or the response message corresponding to the request message is discarded in the bus switching network, it may cause a memory access timeout and generate a memory access exception, which may be, for example, a synchronous external abort (SEA) exception. Normally, after the memory access exception is taken over by the operating system (for example, the operating system kernel), the operating system may end the detection task to eliminate the memory access exception. In order to ensure that the detection task can continue, the detection agent 304 may register an exception handler (for example, an exception handling function) in advance. Through the jump of the exception handler, the detection task can be jumped from the kernel space of the operating system back to the user space, so that the detection agent 304 can continue to execute the detection task.

[0157] Specifically, the detection agent 304 can set an exception jump point during the execution of the detection task, and the context of the detection task can be saved at the exception jump point so that the execution of the detection task can be restored based on the saved context later. When a memory access exception occurs, the detection agent 304 can enter the exception handler, and the exception handler can return to the exception jump point through a function with a cross-function jump function.

[0158] Among them, setting the exception jump point and implementing cross-function jump can be achieved through the setjmp function and longjmp function. Fig. 9 The flowchart of an exception handling process shown in the figure is that the detection agent device 304 includes a detection task execution engine, which registers the SEA exception handler, then obtains the memory address indicated by the detection task, executes the setjmp function, and saves the context of the function here in the jmp_buf structure, so that longjmp can restore the context from this structure. If the function is called directly, the return value is 0; if the function returns from the longjmp call, the return value is non-zero and is provided by the longjmp function. According to the return value of the function, the detection task execution engine can determine whether the setjmp function call is a direct call or a jump from other places.

[0159] When the setjmp function call is a direct call, the detection task execution engine can initiate access to the memory address and wait for a response. When the request message or the response message is discarded, a SEA exception occurs, and the detection task execution engine enters the exception handler to perform exception processing, including: recording the access object where the error occurred, restoring the context through longjmp, returning to the setjump function, and then reporting the fault record to the detection controller 302. After completing the exception processing, the detection task execution engine can continue with the remaining detection tasks.

[0160] S510. The detection agent 304 deployed in the first computing node receives a response message forwarded by the switching node and records the actual detection result.

[0161] It should be noted that Figure 5 The detection access mode is a read mode. When the detection access mode is a write mode, the fault location access of the embodiment of the present application may not execute the above S509 and S510. For example, the second computing node may directly report the fault record when the actual detection result is inconsistent with the expected detection result.

[0162] S512: When the detection agent 304 deployed in the first computing node determines that a fault occurs according to the actual detection result, it reports a fault record.

[0163] When accessing the memory in a read mode, the detection agent device 304 deployed in the first computing node can determine whether a fault has occurred by checking whether the actual detection result read is consistent with the expected detection result. If not, it indicates that a fault has occurred and the first computing node can report a fault record.

[0164] When the memory is detected in a write manner, the detection agent device 304 deployed in the second computing node can check whether the detection memory of the second computing node is rewritten to the expected detection result. If the detection memory is not rewritten successfully, it indicates that a fault has occurred, and the second computing node can report the fault record to the detection controller 302.

[0165] Among them, when the detection task is a repetitive task, for example, a periodic task, the detection agent device 304 deployed in the second computing node can periodically erase the detection memory, and after each detection cycle, check whether the detection memory of the second computing node is rewritten to the expected detection result, so as to determine whether a fault occurs.

[0166] S514 , the detection controller 302 collects statistical data from the statistical agent 306 .

[0167] The detection controller 302 may actively collect statistical data from the switching node (e.g., a statistical agent 306 deployed in the switching node). In some examples, the statistical agent 306 may also actively report statistical data to the detection controller 302. The statistical data may include the number of packets carrying the detection tag received by the switching node and the number of packets carrying the detection tag sent by the switching node.

[0168] like Fig.10As shown, the entire life cycle of the detection task can be divided into a memory detection phase, a statistics collection phase, and a data verification phase. The detection controller 302 can collect statistical data after the memory detection phase ends to ensure that the request message carrying the detection tag has been sent to the destination in the correct case, and ensure that the statistical data collection is complete.

[0169] Specifically, when the memory detection phase ends, the statistical agent 306 can wait for a period of time, which may include the time required for the request message carrying the detection mark sent by the first computing node to reach the second computing node through the bus switching network. Further, the time may also include the time required for the response message sent by the second computing node to reach the first computing node through the bus switching network. In this way, under normal circumstances, the request message and the response message have arrived at the destination node, and the statistical agent 306 can start collecting statistical data.

[0170] S516. The detection controller 302 determines the fault location based on the statistical data.

[0171] The detection controller 302 may trigger fault location when the detection task detects a fault. For example, when the detection controller 302 receives a fault record reported by a computing node (such as the first computing node), it may determine the fault location based on statistical data. Fig.10 As shown, the detection controller 302 can perform verification based on the statistical data after collecting the statistical data, thereby realizing fault location. The detection controller 302 can perform data verification in at least one of the following ways:

[0172] In a first implementation, the detection controller 302 checks whether the number of messages carrying the detection tag received by the same switching node is equal to the number of messages carrying the detection tag sent by the same switching node. When the number of messages carrying the detection tag received by the first switching node in at least one switching node is not equal to the number of messages carrying the detection tag sent by the first switching node, the detection controller 302 determines that the fault location in the bus switching network includes the first switching node.

[0173] In a second implementation, the bus switching network includes a first switching node and a second switching node, and each switching node includes at least one input interface and at least one output interface. The output interface of the first switching node is connected to the input interface of the second switching node. The detection controller 302 checks whether the number of messages carrying the detection mark sent by the output interface of the first switching node and the number of messages carrying the detection mark received by the input interface of the second switching node are equal. When the number of messages carrying the detection mark sent by the first switching node through the output interface is not equal to the number of messages carrying the detection mark received by the second switching node through the input interface, the detection controller 302 determines that the fault location in the bus switching network includes a link from the output interface of the first switching node to the input interface of the second switching node. Among them, the link from the output interface of the first switching node to the input interface of the second switching node may include a network transmission channel from the output interface of the first switching node to the input interface of the second switching node, or the output interface of the first switching node and the input interface of the second switching node.

[0174] When the detection task is a repetitive task, such as a periodic task, the detection controller 302 may subtract the statistical data of the current cycle from the statistical data collected in the previous cycle (previous round) to obtain the incremental data of the current cycle, as shown below:

[0175] C n =C tn -C tn-1 (1)

[0176] Among them, C tn Indicates the statistics of the current period, C tn-1 Indicates the statistics of the previous period, C n Indicates the incremental data of the current cycle.

[0177] Then, the detection controller 302 may perform verification according to the incremental data of the current cycle. The specific verification method may include:

[0178] In the first verification method, the detection controller 302 obtains the number of packets carrying the detection mark received by all interfaces of the switching node according to the incremental data of the current cycle (denoted as ∑ i I i ), and the number of packets carrying detection tags sent by all interfaces of the switching node (denoted as ∑ i O i ), and then determine whether the following formula is satisfied:

[0179] ∑ i I i =∑ i O i (2)

[0180] If not, it means that the switching node has a fault (or anomaly).

[0181] In the second verification method, assuming that the first switching node is connected to the interface b of the second switching node through the interface a, the detection controller 302 can obtain the number of messages carrying the detection mark sent by the interface a of the first switching node (denoted as S out ), and the number of packets carrying the detection tag received by the interface b of the second switching node (denoted as T in ), and then determine whether the following formula is satisfied:

[0182] S out =T in (3)

[0183] If not satisfied, for example, S out >T in , it means that there is a fault (or anomaly) in the link from interface a of the first switching node to interface b of the second switching node.

[0184] In some possible implementations, the detection controller 302 may also obtain expected statistical data of each interface of the switching node (for example, the number of packets carrying the detection mark expected to be received or sent, which may usually be equal to the number of packets carrying the detection mark sent by the source node), and then compare it with the collected actual statistical data to determine the fault location. When determining the fault location, the detection controller 302 may also combine the success or failure of the detection task execution to improve the efficiency and accuracy of fault location.

[0185] Next, the fault location method of the present application is described in conjunction with specific application scenarios.

[0186] See also Fig.11 A schematic diagram of an application scenario of a fault location method is shown. In this example, the detection controller sends a detection task to each computing node in the computing cluster. The detection task instructs each computing node to send a request message carrying a detection tag at a configured detection rate (detection message sending rate). Among them, the detection task also instructs each computing node in each node group to send a request message to the computing node at the same position in other node groups to access the detection memory of the computing node at the same position in other node groups, so as to cover all forwarding paths of the bus switching network and cover the routing table of the switching node in the bus switching network. That is, the first node can be each computing node in the computing cluster, and the second node can be a computing node at the same position in other node groups outside the node group where each computing node is located.

[0187] In the memory detection phase, the computing node turns on the detection function, and each computing node sends at least (Y-1) requests to cover all forwarding paths and the routing table of the switching nodes in the bus switching network. Among them, Y is the number of node groups in the computing cluster. In addition, since the switching nodes select routes randomly when forwarding, the computing nodes can send as many request messages as possible. For example, the number of request messages sent by each computing node can be N*(Y-1), so that all paths can be covered as much as possible.

[0188] The switching nodes in the bus switching network are also configured with matching rules, which may include matching fields and matching feature values ​​for matching messages carrying detection tags. The matching fields may be indicated by masks (also called matching masks). This is described below with an example. Fig.12 As shown, in this example, for a single-bit trace_tag, the matching mask can be used to determine that the matching field is the bit with sequence number 17 in the second row of the message, and the message feature can be used to determine that the matching feature value is 1. Based on this, when the value of the corresponding bit of the message to be matched is 1, it indicates that the match is successful. For a multi-bit trace_tag, the matching mask can be used to determine that the matching field is the bits with sequence numbers 16 to 19 in the second row of the message, and the message feature can be used to determine that the bit with sequence number 16 is used to identify whether the message is a message to be counted, and the feature value corresponding to this bit can be 1, and the bits with sequence numbers 17 to 19 are used to identify the message type, and different values ​​can correspond to different message types. The feature values ​​corresponding to the bits with sequence numbers 17 to 19 can be 010. Based on this, when the values ​​of the bits with sequence numbers 16 to 19 in the second row of the message to be matched are 1010, it indicates that the match is successful, that is, the message is a message of a specific type to be counted.

[0189] like Fig.13 As shown, assuming that after the memory detection phase, the detection controller 302 finds that there is packet loss from computing node 1.4 to computing node 1.8, then all nodes on all possible paths are checked. Assuming that the statistical data of the messages with trace_tag=1 sent and received on interfaces a, b, and c at switching node 3.3 are a: (8, 8), b: (8, 6), and c: (8, 8), wherein x in the statistical data (x, y) represents the number of messages received by the interface, and y represents the number of messages sent by the interface, then the sum of the input and trace_tag=1 message counts is 24, and the sum of the output and trace_tag=1 message counts is 22, and the detection controller 302 can determine that packet loss occurs on switching node 3.3. In other words, the fault location can include switching node 3.3.

[0190] Based on the fault location method of the above embodiment, the present application further provides a detection controller 302. The detection controller 302 is used to execute the fault location method of the above embodiment to locate the fault of the bus switching network in the computing cluster. The detection controller 302 of the embodiment of the present application will be introduced from the perspective of functional modularization.

[0191] See also Fig.14 A schematic diagram of the structure of a detection controller 302 is shown, and the detection controller 302 includes:

[0192] A communication module 3022 is used to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network;

[0193] The communication module 3022 is further used to obtain the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection mark is sent by the first node according to the detection configuration;

[0194] The fault location module 3024 is used to determine the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0195] Exemplarily, the communication module 3022 and the fault location module 3024 may be implemented by hardware or software.

[0196] Among them, when implemented by software, the communication module 3022 and the fault location module 3024 can be applications running on computer devices, such as computing engines, etc. Further, the application can also be virtualized through virtualization services. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, VM services can be services that virtualize virtual machine (VM) resource pools on multiple physical hosts through virtualization technology to provide users with VMs for use on demand. BMS services are services that virtualize BMS resource pools on multiple physical hosts to provide users with BMS for use on demand. Container services are services that virtualize container resource pools on multiple physical hosts to provide users with containers for use on demand. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of traditional physical machines, and it has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0197] When implemented by hardware, the communication module 3022 may be a transceiver, and the fault location module 3024 may include at least one computing device, such as a server, etc. Alternatively, the fault location module 3024 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0198] In some possible implementations, the fault location module 3024 is specifically used to:

[0199] When the number of messages carrying the detection mark received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.

[0200] In some possible implementations, the bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically configured to:

[0201] When the number of packets carrying the detection mark sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection mark received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.

[0202] In some possible implementations, the first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.

[0203] In some possible implementations, the fault location module 3024 is specifically used to:

[0204] The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

[0205] In some possible implementations, the detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.

[0206] In some possible implementations, the number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.

[0207] The above describes the detection controller 302 from the perspective of functional modularization. Next, the detection controller 302 is described from the perspective of hardware entity.

[0208] The present application also provides a detection controller 302. Fig.15As shown, the detection controller 302 includes: a bus 1502, a processor 1504, a memory 1506 and a communication interface 1508. The processor 1504, the memory 1506 and the communication interface 1508 communicate with each other through the bus 1502. It should be understood that the present application does not limit the number of processors and memories in the detection controller 302.

[0209] The bus 1502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 The bus 1502 may include a path for transmitting information between various components of the detection controller 302 (eg, the memory 1506, the processor 1504, and the communication interface 1508).

[0210] The processor 1504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0211] The memory 1506 may include a volatile memory, such as a random access memory (RAM). The memory 1506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to implement the aforementioned fault location method. Specifically, the memory 1506 stores instructions for the detection controller 302 to execute the fault location method.

[0212] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the detection controller 302 and other devices or a communication network.

[0213] This application also provides a computing cluster. Figure 3 As shown, the computing cluster includes multiple computing nodes and a bus switching network for implementing memory access between the multiple computing nodes.

[0214] A first node in the computing cluster, configured to receive a detection configuration sent by the detection controller 302, wherein the detection configuration includes information of a second node detected by the first node, wherein the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network;

[0215] The first node is further configured to send a message carrying a detection mark to the second node according to the detection configuration;

[0216] At least one switching node is used to send to the detection controller the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates the detection to the second node.

[0217] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the detection controller 302 or the computing cluster, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the detection controller 302 or the computing cluster to execute the fault location method.

[0218] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on the detection controller 302, run on a computing cluster, or stored in any available medium. When the computer program product is run on the detection controller 302 or the computing cluster, the detection controller 302 or the computing cluster executes the above-mentioned fault location method.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault location method, It is characterized in that The method comprises: The detection controller sends a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, and the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network; The detection controller obtains the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the messages carrying the detection mark are sent by the first node according to the detection configuration; The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node.

2. The method according to claim 1, It is characterized in that The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: When the number of messages carrying the detection tag received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection tag sent by the first switching node, the detection controller determines that the fault location in the bus switching network includes the first switching node.

3. The method according to claim 1, It is characterized in that The bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the detection controller determines a fault location in the bus switching network according to the number of messages carrying a detection tag received by the at least one switching node and the number of messages carrying a detection tag sent by the at least one switching node, including: When the number of packets carrying the detection tag sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection tag received by the second switching node through the incoming interface, the detection controller determines that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.

4. The method according to any one of claims 1 to 3, It is characterized in that The first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.

5. The method according to any one of claims 1 to 4, It is characterized in that The detection controller determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: The detection controller receives the fault record reported by the first node or the second node, and determines the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

6. The method according to claim 5, It is characterized in that The detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.

7. The method according to any one of claims 1 to 6, It is characterized in that The number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.

8. The method according to any one of claims 1 to 7, It is characterized in that The detection controller is independent of the computing cluster.

9. The method according to any one of claims 1 to 7, It is characterized in that The detection controller is a node in the computing cluster, or the detection controller is deployed on a node in the computing cluster.

10. A fault location method, It is characterized in that Applied to a computing cluster, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the method includes: A first node in the computing cluster receives a detection configuration sent by a detection controller, wherein the detection configuration includes information of a second node detected by the first node, wherein the first node and the second node are any computing node in the plurality of computing nodes or any switching node in the bus switching network; The first node sends a message carrying a detection mark to the second node according to the detection configuration; At least one switching node sends to the detection controller the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.

11. The method according to claim 10, It is characterized in that The first node includes a detection agent device, and the switching node includes a statistical agent device; The first node sends a message carrying a detection mark to the second node according to the detection configuration, including: The detection agent device in the first node sends a message carrying a detection mark to the second node according to the detection configuration; The at least one switching node sends to the detection controller the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node, including: The statistical agent in the at least one switching node sends the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node to the detection controller.

12. The method according to claim 10 or 11, It is characterized in that The first node is any computing node among the plurality of computing nodes, and the detection configuration further includes a detection mode; The method further comprises: When the detection mode is global detection, the first node carries the detection mark in the service message; The message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.

13. The method according to any one of claims 10 to 12, It is characterized in that The detection configuration also includes detecting expected results, and the method further includes: When the first node or the second node detects that the expected detection result is inconsistent with the actual detection result, a fault record is reported to the detection controller, and the fault record is used to trigger the detection controller to locate the fault.

14. The method according to any one of claims 10 to 13, It is characterized in that The number of paths from the first node to the second node is M, where M is a positive integer. The first node sends a message carrying a detection mark to the second node according to the detection configuration, including: The first node sends N messages carrying detection tags to the second node according to the detection configuration, where N is greater than or equal to M.

15. A detection controller, It is characterized in that The detection controller comprises: A communication module, configured to send a detection configuration to a first node in a computing cluster, wherein the detection configuration includes information about a second node detected by the first node, the computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes, the first node and the second node are any computing node among the plurality of computing nodes or any switching node in the bus switching network; The communication module is further used to obtain the number of messages carrying the detection mark received by at least one switching node and the number of messages carrying the detection mark sent by at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node, and the message carrying the detection mark is sent by the first node according to the detection configuration; A fault location module is used to determine the fault location in the bus switching network according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

16. The detection controller according to claim 15, It is characterized in that The fault location module is specifically used for: When the number of messages carrying the detection mark received by a first switching node among the at least one switching node is not equal to the number of messages carrying the detection mark sent by the first switching node, it is determined that the fault location in the bus switching network includes the first switching node.

17. The detection controller according to claim 15, It is characterized in that The bus switching network includes a first switching node and a second switching node, an outbound interface of the first switching node is connected to an inbound interface of the second switching node, and the fault locating module is specifically used for: When the number of packets carrying the detection mark sent by the first switching node through the outgoing interface is not equal to the number of packets carrying the detection mark received by the second switching node through the incoming interface, it is determined that the fault location in the bus switching network includes a link from the outgoing interface of the first switching node to the incoming interface of the second switching node.

18. The detection controller according to any one of claims 15 to 17, It is characterized in that The first node is any computing node among the multiple computing nodes, and the detection configuration also includes a detection mode. When the detection mode is global detection, the service message of the first node carries the detection mark, and the message carrying the detection mark includes a memory detection message and a service message carrying the detection mark.

19. The detection controller according to any one of claims 15 to 18, It is characterized in that The fault location module is specifically used for: The fault record reported by the first node or the second node is received, and the fault location in the bus switching network is determined according to the number of messages carrying the detection mark received by the at least one switching node and the number of messages carrying the detection mark sent by the at least one switching node.

20. The detection controller according to claim 19, It is characterized in that The detection configuration also includes an expected detection result, and the fault record is reported when the first node or the second node detects that the expected detection result is inconsistent with the actual detection result.

21. The detection controller according to any one of claims 15 to 20, It is characterized in that The number of messages carrying detection tags sent by the first node to the second node is greater than or equal to the number of paths from the first node to the second node.

22. A detection controller, It is characterized in that The detection controller comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the detection controller executes the method according to any one of claims 1 to 9.

23. A computing cluster, It is characterized in that The computing cluster includes a plurality of computing nodes and a bus switching network for implementing memory access between the plurality of computing nodes; A first node in the computing cluster is used to receive a detection configuration sent by a detection controller, wherein the detection configuration includes information of a second node detected by the first node, and the first node and the second node are any computing node in the multiple computing nodes or any switching node in the bus switching network; The first node is further configured to send a message carrying a detection mark to the second node according to the detection configuration; At least one switching node is used to send to the detection controller the number of messages carrying the detection tag received by the at least one switching node and the number of messages carrying the detection tag sent by the at least one switching node, wherein the at least one switching node is a switching node in the bus switching network through which the first node initiates detection to the second node.

24. A computer-readable storage medium, It is characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9 or 10 to 14.

25. A computer program product, It is characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9 or 10 to 14.

Citation Information

Cited By

  • Transmission path fault positioning method, system and device and storage medium

    CN119945892A