Processing method, device, equipment, medium and program product of business cluster

Abstract

The embodiment of the present application discloses a processing method, device, equipment, medium and program product for a service cluster. The method includes: obtaining a first communication element group of multiple service flows in a service cluster; obtaining a second communication element group of abnormal service devices in the service cluster; filtering out abnormal service flows from the service cluster based on the first communication element group and the second communication element group; determining the communication path of the abnormal service flow according to the GPU card corresponding to the source end of the abnormal service flow and the GPU card corresponding to the destination end; performing equipment analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow. The use of the embodiment of the present application can significantly reduce the waiting time of the service cluster during abnormal troubleshooting, thereby improving the efficiency of task execution.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of communications, and specifically to a method, device, equipment, medium and program product for processing a service cluster. Background Art

[0002] A business cluster is a network cluster that includes a large number of business devices and network element devices. With the rapid development of science and technology, tasks often require huge computing power support. Therefore, the high-performance network clusters used for task execution often have a huge network scale, a large number of devices and links, resulting in a high frequency of failures and the inability of tasks to run stably for a long time.

[0003] It has been found in practice that when an exception occurs in the execution of a task in a business cluster, the existing mainstream solution is to locate the cause of the exception in the business cluster through manual detection, such as manually collecting the device logs of each business device in the business cluster and issuing alarms for comparative analysis of each device. This results in a long time required to delimit or locate the cause of the exception in the business cluster. The waiting of the business cluster during the exception investigation will cause a large amount of computing power loss, reducing the efficiency of task execution. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, medium and program product for processing a business cluster, which can significantly reduce the waiting time of the business cluster during abnormality troubleshooting, thereby improving task execution efficiency.

[0005] On the one hand, an embodiment of the present application provides a method for processing a service cluster, wherein the service cluster includes multiple service devices and multiple network element devices, and any two service devices are communicatively connected through at least one network element device; each service device includes at least one GPU card; the method includes:

[0006] Obtain a first communication element group of multiple service flows in a service cluster, where a service flow refers to a data flow from a GPU card in a service device at a source end to a GPU card in a service device at a destination end; the first communication element group includes an identifier of a GPU card corresponding to a source end of the service flow and an identifier of a GPU card corresponding to a destination end;

[0007] Acquire a second communication element group of an abnormal service device in the service cluster, where the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card;

[0008] Based on the first communication element group and the second communication element group, screening out abnormal service flows from the service cluster;

[0009] Determine the communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, wherein the communication path includes multiple target network element devices that the abnormal service flow passes through in sequence;

[0010] Perform device analysis and processing on each target network element device on the communication path to obtain abnormal information of abnormal service flows.

[0011] On the other hand, an embodiment of the present application provides a processing device for a service cluster, wherein the service cluster includes multiple service devices and multiple network element devices, and any two service devices are communicatively connected through at least one network element device; each service device includes at least one GPU card; the device includes:

[0012] An acquisition unit is used to acquire a first communication element group of multiple service flows in a service cluster, where a service flow refers to a data flow from a GPU card in a service device at a source end to a GPU card in a service device at a destination end; the first communication element group includes an identifier of a GPU card corresponding to a source end of the service flow and an identifier of a GPU card corresponding to a destination end;

[0013] The acquisition unit is further used to acquire a second communication element group of an abnormal service device in the service cluster, the second communication element group including an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card;

[0014] A processing unit, configured to filter out abnormal service flows from the service cluster based on the first communication element group and the second communication element group;

[0015] The processing unit is further used to determine the communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, wherein the communication path includes multiple target network element devices that the abnormal service flow passes through in sequence;

[0016] The processing unit is also used to perform device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow.

[0017] In one implementation, the service cluster includes X-level networking, the X-level networking includes X-layer subnetworks distributed in a hierarchical manner, each layer of the subnetwork includes multiple network element devices, X is an integer and X≥2; the processing unit is used to determine the communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, specifically for:

[0018] According to the GPU card corresponding to the source end of the abnormal business flow, the GPU card corresponding to the destination end of the abnormal business flow, and the network element devices in each layer of the sub-network in the X-level networking, multiple path groups between the X-layer sub-networks are obtained; each path group includes one or more equal-cost paths, and the two path endpoints of the equal-cost path are respectively network element devices in two adjacent layers of sub-networks;

[0019] According to the flow directions of multiple path groups and abnormal business flows between the X-layer sub-networks, a directed graph is constructed;

[0020] Perform path search processing on the directed graph to obtain the communication path of the abnormal business flow.

[0021] In one implementation, the processing unit is used to obtain multiple path groups between X-layer subnetworks according to the GPU card corresponding to the source end of the abnormal service flow, the GPU card corresponding to the destination end of the abnormal service flow, and the network element device in each layer of the subnetwork in the X-level networking, specifically for:

[0022] According to the identification of the GPU card corresponding to the source end of the abnormal service flow, the starting network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0023] According to the identification of the GPU card corresponding to the destination end of the abnormal service flow, the terminal network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0024] Based on the start network element device and the end network element device, path connection processing is performed in the X-layer sub-network to obtain multiple path groups between the X-layer sub-networks.

[0025] In one implementation, any layer of subnetwork in the X-layer subnetwork is represented as an i-th layer subnetwork, where i is an integer and 1≤i≤X; and a processing unit is configured to perform path connection processing in the X-layer subnetwork based on a starting network element device and an end network element device, and obtain multiple path groups between the X-layer subnetworks, specifically configured to:

[0026] When i=1, according to the starting network element device and the i+1th layer sub-network, multiple first path groups corresponding to the starting network element device are queried according to the border gateway protocol; the opposite end device of each first path group is a network element device in the i+1th layer sub-network;

[0027] Let i=i+1, if i=X, then according to the opposite end device and the terminal network element device of each first path group, query the Xth path group corresponding to the opposite end device of each first path group according to the border gateway protocol; the opposite end device of the Xth path group is the terminal network element device;

[0028] The multiple path groups between the X-layer sub-networks include: multiple first path groups, and X-th path groups corresponding to the opposite-end devices of each first path group.

[0029] In one implementation, the processing unit is further configured to:

[0030] Let i=i+1, if i<X, then according to the peer device of each i-1th path group in the multiple i-1th path groups and the i+1th layer subnetwork, query the multiple i-th path groups corresponding to the peer device of each i-1th path group according to the border gateway protocol;

[0031] When i=X, according to the peer device of each X-1th path group and the X-1th layer subnetwork, query the multiple X-th path groups corresponding to the peer device of each X-1th path group according to the border gateway protocol;

[0032] Let i=i-1, if i>2, then according to the peer device of each 2X-i-1 path group in the multiple 2X-i-1 path groups and the i-1 layer sub-network, query the multiple 2X-i path groups corresponding to the peer device of each i+1 path group according to the border gateway protocol; the peer device of the 2X-i path group is a network element device in the i-1 layer sub-network;

[0033] When i=2, according to the peer device and terminal network element device of each 2X-i-1 path group, query the 2X-2 path group corresponding to the peer device of each 2X-i-1 path group according to the border gateway protocol; the peer device of the 2X-2 path group is the terminal network element device;

[0034] Among them, the multiple path groups between the X-layer subnetworks include: multiple first path groups, multiple i-th path groups, multiple 2X-i path groups, and 2X-2 path groups corresponding to the opposite end devices of the 2X-i path groups.

[0035] In one implementation, the processing unit is used to construct a directed graph according to the flow directions of the multiple path groups and the abnormal service flows between the X-layer sub-networks, specifically to:

[0036] Using multiple network element devices in multiple path groups as nodes of a directed graph;

[0037] According to the flow direction of the abnormal business flow between the X-layer sub-networks, multiple nodes are connected to obtain a directed graph.

[0038] In one implementation, the directed graph includes a start node, an end node, and X-1 intermediate layers, each intermediate layer includes one or more intermediate nodes, and any intermediate layer in the X-1 intermediate layers is represented as the jth intermediate layer, where j is an integer and 1≤j≤X-1; the processing unit is used to perform path search processing on the directed graph, and when obtaining the communication path of the abnormal service flow, it is specifically used to:

[0039] When j=1, starting from the starting node, traverse the paths between the starting node and each intermediate node in the first intermediate layer to obtain multiple first weight values ​​corresponding to the starting node; one first weight value is the weight value of the path between the starting node and an intermediate node in the first intermediate layer;

[0040] Let j=j+1, when j>X-1, calculate the Xth weight value of the path between each intermediate node and the terminal node in the j-1th intermediate layer respectively; an Xth weight value is the weight value of the path between an intermediate node and the terminal node in the j-1th intermediate layer;

[0041] Based on the multiple first weight values ​​and the Xth weight value, a shortest path is determined from the directed graph as a communication path for the abnormal business flow.

[0042] In one implementation, the processing unit is further configured to:

[0043] Let j=j+1, if j≤X-1, then starting from each intermediate node in the j-1th intermediate layer, traverse the paths between each intermediate node in the j-1th intermediate layer and each intermediate node in the j-1th intermediate layer, and obtain multiple jth weight values ​​corresponding to each intermediate node in the j-1th intermediate layer;

[0044] Based on the multiple first weight values, the multiple j-th weight values ​​corresponding to each intermediate node in the j-th intermediate layer and the X-th weight value, the shortest path is determined from the directed graph as the communication path of the abnormal business flow.

[0045] In one implementation, the acquiring unit is used to acquire the second communication element group of the abnormal service device in the service cluster, specifically to:

[0046] Obtain one or more second communication element groups reported by abnormal service devices in the service cluster;

[0047] Perform data deduplication processing on one or more second communication element groups to obtain deduplicated second communication element groups.

[0048] In one implementation, the processing unit is used to perform device analysis and processing on each target network element device on the communication path, and when obtaining abnormal information of the abnormal service flow, is specifically used to:

[0049] Obtain one or more detection indicators; the detection indicators include at least one of the following: a traffic congestion indicator and a congestion degree indicator;

[0050] Using each detection indicator of the one or more detection indicators, performing indicator analysis and processing on each target network element device among the multiple target network element devices, and obtaining indicator information of each target network element device under each detection indicator;

[0051] According to the indicator information of each target network element device under each detection indicator, the abnormal business flow is analyzed abnormally to obtain the abnormal information of the abnormal business flow; the abnormal information of the abnormal business flow is used to describe the abnormal cause of the abnormal business flow.

[0052] In one implementation, the network element device includes a chip; a processing unit, and is further configured to:

[0053] Obtain a reference path, the reference path including: a service flow between a GPU card corresponding to a source end of an abnormal service flow and a GPU card corresponding to a destination end of the abnormal service flow, and multiple reference network element devices passed by the service flow; the reference path is planned by the chip based on the first communication element group of the abnormal service flow;

[0054] Compare multiple reference network element devices on the reference path with multiple target network element devices on the communication path to obtain a comparison result;

[0055] If the comparison result is that the multiple reference network element devices are the same as the multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, then triggering the step of performing device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow;

[0056] If the comparison result is: multiple reference network element devices are different from multiple target network element devices, or the communication connection sequence between multiple reference network element devices is different from the communication connection sequence between multiple target network element devices, the chip abnormality cause analysis is performed to obtain the chip abnormality information.

[0057] In one implementation, the processing unit is further configured to:

[0058] According to the abnormal information of the abnormal business flow, the business cluster is adjusted to obtain an adjusted business cluster;

[0059] Among them, there is no abnormal business flow in the adjusted business cluster.

[0060] In one implementation, the processing unit is further configured to perform cluster adjustment processing on the service cluster according to the abnormal information of the abnormal service flow, and when the adjusted service cluster is obtained, specifically to:

[0061] If the abnormal cause described by the abnormal information of the abnormal business flow is that the GPU card corresponding to the source end of the abnormal business flow is faulty, the GPU card corresponding to the source end of the abnormal business flow is replaced in the business cluster to obtain a replaced business cluster.

[0062] On the other hand, an embodiment of the present application provides a computer device, the device comprising:

[0063] A processor for loading and executing computer programs;

[0064] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processing method of the service cluster is implemented.

[0065] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned service cluster processing method.

[0066] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the above-mentioned service cluster processing method is implemented.

[0067] In an embodiment of the present application, when an abnormality exists in a service cluster, a computer device will obtain a first communication element group of multiple service flows in the service cluster during an abnormal period, and the first communication element group includes an identifier of a GPU card corresponding to the source end of the service flow and an identifier of a GPU card corresponding to the destination end. At the same time, the computer device also obtains a second communication element group of abnormal service devices in the service cluster, and the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card. Since the service flows generated by all service devices that communicate in the service cluster during the abnormal period are collected, the computer device can filter out abnormal service flows with abnormalities from all service flows in the service cluster based on the first communication element group of service flows generated by all service devices and the second communication element group generated by the abnormal service devices; the troubleshooting scope for all service devices is narrowed to the troubleshooting scope for abnormal service flows, which greatly shortens the time spent on abnormal cause analysis. Further, the computer device can restore the communication path composed of multiple target network element devices that the abnormal business flow passes through in sequence from the business cluster according to the GPU card corresponding to the source end of the abnormal business flow and the GPU card corresponding to the destination end; compared with the need to perform device analysis on each business device in the business cluster, the scope of troubleshooting is further narrowed to multiple target network element devices included in the communication path for the abnormal business flow, and the computer device only needs to perform device analysis and processing on multiple target network element devices, and can quickly analyze and obtain abnormal information of the abnormal business flow. It can be seen that the embodiment of the present application effectively restores the communication path of the abnormal business flow when the business cluster is abnormal, and compared with the analysis of all business devices in the business cluster when the fault occurs, the fault troubleshooting scope domain (or fault domain) to be investigated is reduced from all business devices participating in the task in the business cluster to only abnormal business devices that have a communication relationship and upload the second communication element group during the abnormal time period, specifically the communication path of the abnormal business flow generated by the abnormal business device (i.e., the communication path of the root cause GPU card); the processing efficiency of faults in the business cluster is greatly improved, while avoiding the waste of computing power of business devices in the business cluster, saving the cost of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1a It is a schematic diagram of the network architecture of a two-level network;

[0070] Figure 1b It is a schematic diagram of a three-level network architecture;

[0071] Figure 2a It is a flowchart of troubleshooting for an existing mainstream manual collection task;

[0072] Figure 2b It is a flowchart of a troubleshooting provided by an exemplary embodiment of the present application;

[0073] Figure 3 It is a schematic diagram of the architecture of a processing system of a service cluster provided by an exemplary embodiment of the present application;

[0074] Figure 4 It is a flowchart of a method for processing a service cluster provided by an exemplary embodiment of the present application;

[0075] Figure 5 It is a schematic diagram of comparing a first communication element group and a second communication element group provided by an exemplary embodiment of the present application;

[0076] Figure 6a It is a schematic diagram of calling a data preprocessing module to construct a directed graph provided by an exemplary embodiment of the present application;

[0077] Figure 6b It is a flow chart of a communication path for calling a data preprocessing module to restore an abnormal service flow provided by an exemplary embodiment of the present application;

[0078] Figure 7 It is a schematic diagram of determining multiple path groups between X-layer subnetworks in a two-level networking provided by an exemplary embodiment of the present application;

[0079] Figure 8 It is a schematic diagram of determining multiple path groups between X-layer subnetworks in a three-level networking provided by an exemplary embodiment of the present application;

[0080] Fig. 9 is a schematic diagram of a directed graph provided by an exemplary embodiment of the present application;

[0081] Fig.10a It is a schematic diagram of determining the shortest path in a directed graph corresponding to a secondary network provided by an exemplary embodiment of the present application;

[0082] Fig.10b It is a schematic diagram of determining the shortest path in a directed graph corresponding to a three-level network provided by an exemplary embodiment of the present application;

[0083] Fig.11It is a schematic diagram of an overall process of quickly recovering a communication path of an abnormal service flow through an exemplary embodiment of the present application;

[0084] Fig.12 It is a flowchart of another service cluster processing method provided by an exemplary embodiment of the present application;

[0085] Fig.13a It is a schematic diagram of finding an end-to-end path of an abnormal service flow in a directed graph for visual display provided by an exemplary embodiment of the present application;

[0086] Fig.13b It is a schematic diagram for visually displaying an end-to-end path of an abnormal service flow that is not found in a directed graph, provided by an exemplary embodiment of the present application;

[0087] Fig.14 It is a schematic diagram of a result of performing device analysis and processing on a target network element device on an end-to-end path of an abnormal service flow using a traffic congestion indicator (Packet FlowContex, PFC) and a congestion level indicator (Explicit Congestion Notification, ECN) provided by an exemplary embodiment of the present application;

[0088] Fig.15 It is a structural schematic diagram of a processing device of a service cluster provided by an exemplary embodiment of the present application;

[0089] Fig.16 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0091] In an embodiment of the present application, a processing solution for a business cluster is provided; this solution can quickly locate abnormal business flows that may have abnormalities when an abnormality exists in the business cluster, and restore the communication path of the abnormal business flow. It is only necessary to analyze the cause of the abnormality for the communication path of the abnormal business flow, which greatly shortens the troubleshooting time and improves the troubleshooting efficiency, thereby improving the execution efficiency of tasks in the business cluster.

[0092] A business cluster refers to a high-performance computing system that connects multiple servers in a local area network or the Internet, implements resource sharing and task allocation among multiple servers through software or hardware, and jointly completes tasks in the same business. Among them, business refers to a series of activities carried out to achieve a certain goal, and tasks are specific activity content or goals; business can be regarded as a collection of multiple tasks, each of which is part of the business. In a business cluster, it supports the allocation of multiple tasks of a business to different servers for independent processing, and collects the intermediate results obtained by each server for the execution of different tasks, summarizes the intermediate results, and obtains the execution results of the business. Depending on the business field, the business clusters used to implement the business are different; business clusters may include but are not limited to: model training clusters corresponding to model training business, data processing clusters corresponding to data processing business, and resource processing clusters corresponding to resource management business, etc.; in the embodiment of the present application, the business cluster is taken as a model training cluster as an example to explain the processing scheme of the business cluster, which is specially explained here.

[0093] The service cluster includes: multiple service devices used to perform tasks (such as model training tasks performed in the model training cluster) and multiple network element devices that provide communication between service devices; any two service devices in the service cluster communicate with each other through at least one network element device. The following introduces the concepts of service devices and network element devices in the service cluster, including:

[0094] (1) Business equipment.

[0095] The business equipment is the server mentioned above for performing tasks. When the business cluster is a model training cluster, the business equipment in the business cluster is a high-performance computer server used for model training; these servers usually have high-performance computing capabilities, large-capacity storage space and high-speed network connections, and can meet the needs of large-scale machine learning and deep learning tasks. Among them, each business device in the business cluster includes or is configured with at least one GPU (Graphics Processing Unit) network card (or simply referred to as a GPU card); the GPU network card configured in the business device is a key component for the business device to perform tasks, such as accelerating the execution of computing-intensive tasks, realizing image recognition, target detection and image tasks, and parallel processing of text data to achieve natural language processing. The execution of tasks by business devices mainly depends on the GPU network card deployed in the business device, so the business device can also be called a GPU device.

[0096] (2) Network element equipment.

[0097] A network element device is a network device used to transmit information in a service cluster; network element devices include but are not limited to: servers, routers, switches or other devices with information transmission capabilities. In an embodiment of the present application, it supports the division of network element devices into levels or layers, and network element devices of different levels can be interconnected to transmit information between service devices in a service cluster. An embodiment of the present application supports dividing a network after multiple network element devices in a service cluster into levels as X-level networking, where the X-level networking includes X-layer subnetworks distributed in a hierarchical manner, and each layer of subnetwork includes multiple network element devices; X is an integer and X≥2. The hierarchical distribution between X-layer subnetworks refers to the X-layer subnetworks being arranged in multiple layers in sequence according to the division dimension (such as the functional dimension); each layer of subnetwork undertakes a specific function, thereby building an efficient and reliable network system.

[0098] Any two service devices in the service cluster are connected in communication through at least one network element device, specifically, through X-level networking. In an embodiment of the present application, the data transmitted when any two service devices in the service cluster are connected in communication through X-level networking is called a service flow, and the service flow can be understood as a set of ordered byte data sequences with a starting point and an end point. The source end (source, abbreviated as src) and the destination end (Destination, abbreviated as dst) of the service flow transmitted between two service devices in the service cluster are both GPU network cards integrated in the service device. Considering that the source end and the destination end of the service flow are both GPU network cards integrated in the service device, the service flow transmitted when any two service devices in the service cluster communicate through X-level networking can also be understood as: a data flow from a GPU network card in the service device at the source end to a GPU network card in the service device at the destination end. Among them, the side that sends the service flow in the service flow transmission is called the source end, and the side that receives the service flow is called the destination end.

[0099] Among them, the service flow is transmitted in the service cluster according to the communication element group. As mentioned above, the service flow is essentially a byte data sequence, which includes a communication element group of the service flow, and the communication element group includes multiple fields related to the transmission of the service flow. The communication element group includes a five-tuple, and the five-tuple of the service flow includes: the identification of the GPU card that sends the service flow in the service device at the source end, the identification of the GPU card that receives the service flow in the service device at the destination end, the port number of the port used by the GPU card that sends the service flow in the service device at the source end, the port number of the port used by the GPU card that receives the service flow in the service device at the destination end, and the protocol number of the communication protocol used by the service device at the source end and the service device at the destination end for data transmission. Further, according to the number of elements or fields included in the communication element group, the communication element group can also include a triple or a binary group. The triple includes: the identification of the GPU card that sends the service flow in the service device at the source end, the identification of the GPU card that receives the service flow in the service device at the destination end, and the port number of the port used by the GPU card that sends the service flow in the service device at the source end. The tuple (which can be expressed as (SIP / DIP), SIP is the identifier of the GPU card corresponding to the source end, and DIP is the identifier of the GPU card corresponding to the destination end) includes: the identifier of the GPU card that sends the service flow in the service equipment of the source end, and the identifier of the GPU card that receives the service flow in the service equipment of the destination end. For the convenience of explanation, the GPU network card that sends the service flow can be called the source GPU network card, and the identifier of the source GPU network card is called the source GPU card identifier; the GPU network card that receives the service flow is called the destination GPU network card, and the identifier of the destination GPU network card is called the destination GPU card identifier; the port used by the GPU card that sends the service flow is called the source port, and the port number is called the source port number; the port of the GPU card that receives the service flow is called the destination port, and the port number is called the destination port number.

[0100] In an embodiment of the present application, the business devices in the business cluster transmit business flows through direct memory access technology (Remote Direct Memory Access, RDMA). Direct memory access technology is a network communication technology that allows business devices in a business cluster to directly access the memory of other business devices through network element devices without going through the operating system deployed in the business devices. In other words, between any two business devices in the business cluster, business flows are allowed to be directly transmitted from the memory of one business device to the memory of another business device through the network element device, and the transmission of business flows does not require the intervention of the operating systems of the two business devices. Since the business flow transmission bypasses the operating system of the business device, the business flow transmission speed in the business cluster is significantly improved, the network delay is reduced, and the burden on the central processing unit (CPU) and the operating system of the business device is reduced.

[0101] Based on the above description, it can be seen that the complexity of the network architecture of the service cluster varies depending on the value of X. For example, when X=2, the network architecture diagram of the secondary network in the service cluster can be seen in Figure 1a ;like Figure 1a As shown, the secondary network includes two layers of subnetworks, namely the first layer subnetwork and the second layer subnetwork, and each layer of subnetwork contains network element devices. Among them, the network element devices in the first layer subnetwork are used to communicate and connect directly with the service devices in the service cluster, and to communicate and connect with the network element devices in the second layer subnetwork, so that the service flow issued by the service device can be transmitted to the network via the network element devices in the first layer subnetwork; the network element devices in the second layer subnetwork are used to communicate and connect with the network element devices in the first layer subnetwork, so as to transmit the service flow in the network. In an embodiment of the present application, the network element devices in the first layer subnetwork are called GPULA (GPU Load Advisor, GPU load guidance) devices, and the network element devices in the second layer subnetwork are called GPULC (GPU Load Controller, GPU load control) devices; GPULA devices and GPULC devices work together to help the service cluster better manage resources, optimize cluster performance, and ensure the smooth execution of tasks in the service cluster.

[0102] For example, when X=3, the network architecture diagram of the three-level network in the business cluster can be seen in Figure 1b ;like Figure 1b As shown, the three-level network includes three layers of subnetworks, namely the first layer subnetwork, the second layer subnetwork and the third layer subnetwork, and each layer of subnetwork contains network element devices. Among them, the relevant contents of the first layer subnetwork and the second layer subnetwork are the same as the first layer subnetwork and the second layer subnetwork in the second-level network, and are not repeated here. The three-level network also includes a third layer subnetwork, and the network element device in the third layer subnetwork is called an SGLC (Super GPULC, Super GPU load control) device; the network element device in the third layer subnetwork is used to communicate with the network element device in the second layer subnetwork to realize the transmission of service flows in the network.

[0103] For the sake of convenience, the following takes the X-level networking in the service cluster as a secondary networking or a tertiary networking as an example to introduce the processing scheme of the service cluster provided by the embodiment of the present application, which is specially explained here. Figure 1aTake the secondary network shown as an example; when the service device 1001 in the service cluster needs to send information to the service device 1002, the service device 1001 sends the service flow to a network element device GPULA in the first layer subnet in the secondary network through a GPU network card deployed inside it through a port. The network element device GPULA performs routing selection according to a routing protocol (such as Border Gateway Protocol (BGP)), specifically selecting the network element device GPULA and one or more network element devices GPULC in the second layer subnet for communication connection. The network element device GPULC in the second layer subnet that receives the service flow determines the network element device GPULA in the first layer subnet that can receive the service flow according to the destination port field in the service flow. The network element device GPULA then transmits the service flow to the service device 1002 in the service cluster to realize the service flow transmission between the service device 1001 and the service device 1002.

[0104] In actual applications, the operation and maintenance personnel of the service cluster can only know the network element devices in the first-layer sub-network that communicate with a GPU network card in the service equipment, and cannot perceive the communication network paths or links between the network element devices of adjacent layers within the X-level network. In other words, for the operation and maintenance personnel, the X-level network within the service cluster is basically in a black box state. When the service cluster is abnormal (such as stopping the execution of training tasks due to a fault, or the task execution speed is significantly reduced, or the execution result has a large deviation, etc.), the operation and maintenance personnel cannot perceive the communication network path between the X-level networks. They need to manually collect the device logs of each service device and issue alarms to analyze the cause of the abnormality by comparing and analyzing each service device, which consumes a lot of time. For example, after the model training task is interrupted, it takes more than 24 hours, resulting in a large loss of computing power of the service equipment during the investigation of the abnormal cause, wasting computing power resources and reducing the efficiency of task execution.

[0105] In order to shorten the troubleshooting time in the business cluster and improve the efficiency of task execution, the processing solution for the business cluster provided in the embodiment of the present application aims to automatically calculate and restore the communication path between the GPU network cards in the business cluster through the network flow control technology (Solution for Network Monitoring, sflow). The processing solution for the business cluster is specifically to collect the RDMA communication data (such as the communication element group) on the business cluster during the abnormal time period when the abnormality exists through the sflow agent in the business cluster, pre-process the RDMA communication data into a directed graph structure, and further perform a path search on the directed graph (such as using the depth-first traversal search algorithm in graph theory) to find the real communication network path (or transmission path or communication path, etc.) between the GPU network cards in the business cluster for the abnormal business flow. In this way, the communication path of the abnormal business flow can be restored from the business cluster that was originally in a black box state, and further abnormal cause analysis and troubleshooting can be performed based on the communication path, effectively overcoming the low utilization of business equipment caused by the interruption of task execution (such as the execution of AI (Artificial Intelligence) model training tasks) caused by the GPU network card, and significantly improving the utilization rate of business equipment and computing power. For example, the time for business equipment to wait for troubleshooting is shortened from 24 hours to less than 1 minute, greatly improving the execution efficiency of tasks and reducing the waste of resources and costs caused by task interruptions.

[0106] In the specific implementation, an sflow agent is deployed in all or part of the network element devices in the service cluster; the sflow technology is a network traffic detection technology based on message sampling in accordance with the IEEE (Institute of Electrical and Electronics Engineers) standard; by setting the sampling rate on the network element device, the sflow agent collects messages of the service flow flowing through the network element device according to the set sampling rate. The message is a transmitted data unit, which mainly refers to the first communication element group of the service flow in the embodiment of the present application. In this way, when there is an abnormal situation in the service cluster, the sflow agent in the network element device in the service cluster collects the first communication element group of multiple service flows within the abnormal time period, and reports the first communication element group of each service flow in the multiple service flows to the computer device. At the same time, a TCCL (Tencent Collective Communication Library) module is deployed in the business equipment in the business cluster. The TCCL module supports efficient data transmission between GPU cards through heterogeneous parallel communication technology. When an abnormality occurs in a business device, the TCCL module in the business device reports a second communication element group of the abnormal business device to the computer device. The second communication element group includes the identifier of the abnormal GPU card in the abnormal business device and the identifier of the GPU card that communicates with the abnormal GPU card.

[0107] Since the service flows generated by all service devices that communicate in the service cluster during the abnormal period are collected by sflow, the full service flows collected by sflow include the abnormal service flows sent by the abnormal service devices; therefore, the computer device can filter out the abnormal service flows with abnormalities from all service flows in the service cluster based on the first communication element group of the service flows generated by all service devices and the second communication element group generated by the abnormal service devices. The computer device can restore the communication path of the abnormal service flow in the X-level networking according to the GPU card corresponding to the source end of the abnormal service flow and the GPU card corresponding to the destination end, and can narrow the scope of troubleshooting from all service devices to the communication path between the source end and the destination end of the abnormal service flow. The computer device performs device analysis and processing on multiple target network element devices included in the communication path of the abnormal service flow, and can determine the abnormal information of the abnormal service flow, which is a paragraph describing the reason why the abnormal service flow is abnormal, such as network reasons, target network element device failure or GPU card failure.

[0108] It has been found in practice that the service cluster processing solution provided by the embodiment of the present application has obvious advantages when analyzing the cause of service cluster abnormalities. The following is an example of comparing the present application solution with the existing mainstream service cluster abnormality analysis solution to illustrate the advantages of the embodiment of the present application, wherein:

[0109] The current mainstream solution for analyzing the cause of business cluster anomalies is to analyze and solve the problem by manually collecting alarm logs of all business devices involved in task execution. Figure 2aAs shown in the figure, the process of analyzing the cause of service cluster abnormality usually includes the following steps: collecting device logs, analyzing alarm information, comparing each device, and demarcating and locating. The specific implementation process of each step is as follows: ① Collecting device logs: The operation and maintenance personnel need to log in to each service device in the service cluster and collect log files related to each service device. This collection process needs to be implemented manually, which may involve a large number of service devices and is very time-consuming. ② Analyzing alarm information: Alarm information usually contains key information about the operating status of service devices. However, since alarm information may come from different systems and devices, manual comparison and analysis are required to determine certain devices or systems that have caused problems. ③ Comparison of service devices: The operation and maintenance personnel need to compare the logs and performance indicators of multiple service devices. Specifically, they need to write scripts for careful comparison and analysis to locate network or service device / network element device problems and find out the abnormal service devices that are the root cause. ④ Demarcation and locating: After determining the possible abnormal service devices, the operation and maintenance personnel need to further analyze to determine the specific cause of the problem (such as the optical module of the service device, PFC back pressure, etc.). This demarcation and locating process may involve code review, hardware detection and other aspects. It can be seen that when there is an abnormality in the business cluster, such as when the model training cluster is interrupted, the business equipment will generate corresponding error logs. At this time, the operation and maintenance personnel need to manually log in to each business device and collect the corresponding log files to delimit and locate the abnormal business equipment; this causes an ordinary task execution interruption in the business cluster to usually take more than 24 hours. During this period, the business equipment and other computing resources are in a waiting state, resulting in a large amount of computing power loss, which affects the current task execution and may also affect other projects that rely on these business equipment to execute (such as other businesses that rely on the business cluster to execute tasks). The existing mainstream business cluster abnormality cause analysis scheme has the following disadvantages: ① Time-consuming: As mentioned above, due to the black box nature of the communication path between the X-level networks, the operation and maintenance personnel need to collect the logs of all business devices involved in the task execution, involving a large number of business devices, resulting in hours or even days from the discovery of the abnormality to the final location and solution of the problem; during the waiting period, the business equipment and other computing resources are idle, wasting the computing power of the business equipment and reducing the efficiency of task execution. ② High labor cost: Professional operation and maintenance personnel are required to write scripts to manually analyze and compare the indicator information of all business equipment to find possible abnormal business equipment, which increases labor costs. ③ High risk of error: Misjudgment or omissions may occur during manual operation and analysis, resulting in the problem not being completely resolved.

[0110] The processing scheme of the service cluster provided in the embodiment of the present application is to use the original sflow in the service cluster to automatically collect the first communication element group of the service flow in the service cluster within the abnormal time period, and to use the TCCL module deployed in the service device to automatically report the second communication element group (the schematic diagram of the TCCL module reporting the second communication element group can be seen in Figure 2b ), to automatically locate the abnormal business flow, and restore the communication path of the abnormal business flow in the business cluster according to the abnormal business flow; on the one hand, there is no need to manually collect log information, which improves the efficiency of abnormal troubleshooting and greatly reduces the cost of abnormal troubleshooting. On the other hand, the scope of faults that operation and maintenance personnel need to troubleshoot is reduced from all business devices that have communication in the business cluster when the abnormality occurs to the communication path of the abnormal business flow. Operation and maintenance personnel only need to pay attention to the communication path of the abnormal business flow. The overall troubleshooting time is reduced from days (such as days) to minutes (such as minutes), which can quickly locate abnormal problems and achieve rapid recovery of tasks, thereby improving task execution efficiency.

[0111] The processing scheme of the service cluster proposed in the embodiment of the present application is deployed in a computer device, which is an independent device independent of the service cluster and used to analyze the abnormal causes of the service cluster. When an abnormal situation occurs in the service cluster, the sflow agent in the service cluster reports the first communication element group of the full service flow in the service cluster to the computer device, and the TCCL module in the abnormal service device reports the second communication element group to the computer device; after receiving the first communication element group of the full service flow and the second communication element group of the abnormal service device, the computer device restores the communication path of the abnormal service flow according to the specific implementation logic of the processing scheme of the service cluster, and further combines the basic detection module and other devices to perform device analysis and processing on multiple target network element devices in the communication path to define whether it is a network problem or a device problem, such as which problems of the abnormal service device cause the abnormality, etc. Among them, the basic detection module is a dedicated module or device with device detection capabilities.

[0112] To facilitate understanding of the processing solutions for the business clusters mentioned above, Figure 3 The schematic diagram of the scene shown in the figure briefly introduces the processing system involved in the embodiment of the present application. Figure 3As shown, the system includes a service cluster, and the service cluster includes multiple service devices, such as service device 301 and service device 302, etc. The service cluster also includes X-level networking, which is a two-level networking, and each layer of sub-network in the two-level networking includes multiple network element devices, such as the first layer sub-network includes network element device 303 and network element device 304, and the second layer sub-network includes network element device 305, network element device 306 and network element device 307, etc. The system also includes a computer device 308, which can be a terminal or a server. The embodiment of the present application does not limit the number of service devices and network element devices in the system, nor does it limit the device type and number of computer devices. For example, the computer device can be a separate server and / or terminal, or the computer device is a single server and / or terminal in a distributed network, etc.

[0113] Among them: ① The computer device is a terminal, and the terminal may include but is not limited to: smartphones (such as smartphones with Android system, or smartphones with Internetworking Operating System (IOS)), tablet computers, portable personal computers, mobile Internet devices (MID), vehicle-mounted devices, head-mounted devices, intelligent chat robots and aircraft and other terminal devices. The embodiments of this application do not limit the type of terminal devices, which is explained here. ② The computer device is a server. The server can be an independent physical server, or a server cluster or distributed network (or distributed system) composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. ③ The computer device includes a terminal and a server, that is, the service cluster processing solution provided in the embodiment of the present application is jointly executed by the terminal and the server. The server is used to implement the background logic of the service cluster processing solution. The terminal can output data visually to the operation and maintenance personnel, such as outputting the second communication element group reported by the TCCL module to the operation and maintenance personnel, or outputting the communication path of the abnormal service flow to the operation and maintenance personnel, etc. Through the visual analysis process display, it is greatly facilitated for the operation and maintenance personnel to troubleshoot the service cluster.

[0114] It should be noted that the devices in the service cluster (such as service devices or network element devices) and the computer devices can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0115] Combine the following Figure 3 The secondary networking shown briefly introduces the process of the processing solution of the service cluster provided in the embodiment of the present application. In the specific implementation, it is assumed that the service cluster is a model training cluster; the service flow is transmitted between the service device 301 and the service device 302 in the model training cluster to realize the AI ​​model training task. If an abnormality occurs in the model training cluster during a certain time period (referred to as an abnormal time period or abnormal period) when the model training task is executed, such as when an abnormality occurs in the service device 301 in the model training cluster, an abnormality also occurs in the service device 302 that communicates with the service device 301, thereby slowing down the training speed of the entire model training cluster, then the sflow agent deployed in the network element devices in the model training cluster (such as all or part of the network element devices in the first layer subnet and the second layer subnet, such as network element devices 303, network element devices 304, network element devices 305, network element devices 306, and network element devices 307) will collect the first communication element group of each service flow in the full amount of service flows (or referred to as multiple service flows) between all service devices that communicate in the model training cluster during the abnormal time period, and send the first communication element group of the full amount of service flows to the computer device 308. At the same time, the TCCL module in the abnormal service device 301 in the model training cluster will also report the second communication element group generated by the abnormal service device 301 to the computer device 308.

[0116] After acquiring the first communication element group of multiple business flows and the second communication element group of abnormal business devices, the computer device 308 determines a third communication element group in which the identification of the GPU card corresponding to the source end and the identification of the GPU card corresponding to the destination end are the same based on the first communication element group and the second communication element group. The third communication element group is used as the communication element group of the abnormal business flow. Then, based on the third communication element group, the computer device restores the communication path of the abnormal business flow from the model training cluster, and the communication path includes multiple target network element devices that the abnormal business flow abnormally passes through in the secondary network; if the communication path is Figure 3As shown, GPULA1→GPULC1→GPULA2, GPULA1, GPULC1 and GPULA2 are target network element devices. In this way, the computer device only needs to perform device analysis and processing on the communication path of the abnormal business flow, that is, the abnormal information of the abnormal business flow can be obtained, and the abnormal information describes the abnormal cause of the abnormal business flow in the model training cluster; in detail, the computer device calls the basic detection module to perform indicator analysis on each target network element device in the communication path, and based on the indicator information obtained by the analysis (such as GPU utilization, RDMA traffic and ECN messages, etc.), the network status correlation analysis is performed hop by hop based on the devices and interfaces of the communication path to locate whether it is a network or device problem, effectively reducing the overall troubleshooting time from one day to minutes, and at the same time linking the business side to quickly resume the execution of the model training task to improve the task execution efficiency.

[0117] It can be seen that the embodiment of the present application introduces a processing solution for integrating the computer device 308 with the business cluster. In this way, when an abnormality occurs in the business cluster, the computer device 308 may automatically analyze the cause of the abnormality in the business cluster. Compared with the existing mainstream solution that requires operation and maintenance personnel to manually maintain the business cluster, it greatly saves labor costs and time costs, and intelligent abnormal cause analysis can improve the accuracy of abnormal cause analysis, and using computer equipment to perform abnormal cause analysis shortens the troubleshooting time, thereby improving the efficiency of task execution in the business cluster.

[0118] Based on the above brief introduction to the processing solution and processing system of the service cluster provided in the embodiment of the present application, the following points need to be explained:

[0119] ① The above-mentioned embodiments of the present application Figure 3 The system architecture shown is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. A person skilled in the art will know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems. In other words, Figure 3 The schematic diagram of the processing system shown is an exemplary schematic diagram; in actual applications, the number and type of devices included in the processing system may vary, and the present application embodiment does not limit the schematic diagram of the processing system architecture. Figure 3 The number of computer devices 308 in the system shown may be multiple, and multiple computer devices form a distributed system. Through the distributed system, the specific implementation process of the business cluster processing solution provided by the embodiment of the present application is distributed to different computer devices for execution, thereby effectively improving the execution efficiency of the business cluster processing solution.

[0120] ② In addition to supporting the use of sflow agents to collect the first communication element group to restore the communication path of abnormal business flows, the embodiments of the present application also support the use of other methods to obtain path information between X-layer subnetworks. For example, INT (Inband-Network Telemetry) can be used to collect path information between X-layer subnetworks for each pair of communication sessions; INT technology mainly uses an additional server to send a detection flow, which is used in the business cluster to follow the implementation path of each business flow, thereby achieving the acquisition of path information of the business flow.

[0121] ③ The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as when a computer device obtains the first communication element group and the second communication element group, it is necessary to obtain the permission or consent of the operation and maintenance personnel, and the collection, use and processing of relevant data (such as the analysis process of the abnormal causes in the business cluster, etc.) need to comply with the relevant laws, regulations and standards of the relevant regions.

[0122] Based on the processing scheme of the service cluster described above, the embodiment of the present application proposes a more detailed processing method of the service cluster. The processing method of the service cluster proposed in the embodiment of the present application will be described in detail below with reference to the accompanying drawings. Figure 4 A flowchart of a method for processing a service cluster provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, which is the aforementioned Figure 3 The computer device 308 shown is, for example, a terminal or a server. The processing method of the service cluster may include but is not limited to steps S401-S405:

[0123] S401: Acquire a first communication element group of multiple service flows in a service cluster.

[0124] As mentioned above, a business flow refers to a data flow from a GPU card in a business device in a business cluster to a GPU card in another business device; the first communication element group of a business flow refers to the five-tuple of the business flow, and the first communication element group of any business flow can be expressed as: (source GPU card identifier, destination GPU card identifier, source port number, destination port number, protocol number). In the case where the communication path between the X-level networking in the business cluster is a black box, the computer device can locate the two ends of the business flow in the business cluster through the first communication element group of the business flow, and provide guidance for the subsequent exploration of the communication path of the business flow in the X-level networking, which is conducive to the subsequent restoration of the communication path of the business flow in the X-level networking.

[0125] In a specific implementation, when an exception occurs in the execution of a task in a service cluster, the sflow agent deployed in the network element device of the service cluster collects the first communication element group of each service flow in multiple service flows, and sends the first communication element group of each service flow to the computer device, and the computer device obtains the first communication element group of each service flow in the abnormal time period in the service cluster. Among them, the first communication element group of the service flow collected by the sflow agent includes: the first communication element group of the service flow generated on all service devices participating in the task execution and communicating in the service cluster during the abnormal time period when the abnormality occurs in the service cluster. When an abnormality occurs in the service cluster, the computer device only needs to obtain the first communication element group of the full amount of service flows in the abnormal time period. Compared with collecting the service flows in the entire time period of task execution (i.e., the time period from the start of task execution to the end of execution), the first communication element group of the service flows in the abnormal time period is collected in a targeted manner, which not only reduces the amount of data collection, but also avoids analyzing the service flows generated when the task is executed normally, thereby improving the efficiency of data analysis.

[0126] S402: Acquire a second communication element group of abnormal service devices in the service cluster.

[0127] As mentioned above, a TCCL module is deployed in the service equipment in the service cluster; when a service equipment in the service cluster generates an abnormal interruption, an alarm message will appear in the implementation of TCCL in the service equipment, and the alarm message includes the second communication element group generated in the service equipment with the abnormality; the alarm message can be expressed as error content (ERR CQE), and the second communication element group is the aforementioned triplet, and the triplet is expressed as: (source GPU card identifier, destination GPU card identifier, source port number). Among them, the service equipment with the abnormality is referred to as an abnormal service equipment in the embodiment of the present application, and the TCCL module in the abnormal service equipment will report to the computer equipment all the second communication element groups affected within the abnormal time period; the second communication element group includes the identifier of the abnormal GPU card with the abnormality in the abnormal service equipment, and the identifier of the GPU card communicating with the abnormal GPU card, and the GPU card communicating with the abnormal GPU card here is the GPU card that will be affected by the abnormal GPU card.

[0128] In a service cluster, service devices often include multiple GPU cards. For example, a service device includes 8 GPU cards. The multiple GPU cards communicate with each other through a hardware bus (PCI (Peripheral Component Interconnect) or NVLink (NVIDIA NVLink, a high-speed bus and communication protocol), etc.), rather than through X-level networking. Based on this, when a GPU card in a service device is abnormal (the GPU card is called an abnormal GPU card), in addition to the GPU cards on other service devices that communicate with the abnormal GPU card through X-level networking, other GPU network cards deployed on the same service device as the abnormal GPU card will also be affected, so the other GPU network cards will also report the computer device abnormality. It can be seen that the number of second communication element groups reported by the abnormal service device in the service cluster obtained by the computer device is multiple, and the multiple second communication element groups include: a second communication element group of service flows that communicate through X-level networking, and a second communication element group of hardware communication that communicates through a hardware bus.

[0129] Furthermore, the second communication element group of the service flow communicating through the X-level networking includes the second communication element group generated by the root cause GPU card and the second communication element group generated by the non-root cause GPU card affected by the root cause GPU card; that is, in the service cluster, not only the abnormal service device that directly generates the abnormality reports the second communication element group, but also the abnormal service device affected by the abnormality reports the second communication element group. For example, the abnormality is caused by the failure of the root cause GPU card in service node A, and the root cause GPU card and the non-root cause GPU card in service node B communicate through the X-level networking. Then, not only the TCCL in service node A will report the second communication element group generated by the root cause GPU card, but the TCCL in service node B will also report the second communication element group generated by the non-root cause GPU card. It can be seen from this that the root cause GPU card refers to the GPU card that directly generates the abnormality or is directly affected by the failure (or the GPU card that is directly affected by the abnormal device (such as the network element device, etc., or the GPU card with a fault, etc.)), and the non-root cause GPU card refers to the GPU card that is affected by the root cause GPU card and causes the task execution speed to decrease or the task to stop executing.

[0130] S403: Filter out abnormal service flows from the service cluster based on the first communication element group and the second communication element group.

[0131] As mentioned in the aforementioned step S402, the multiple second communication element groups acquired by the computer device include not only the second communication element group of the business flow communicating through the X-level networking, but also the second communication element group of the hardware communication communicating through the hardware bus, and the second communication element group of the hardware communication communicating through the hardware bus will not have an end-to-end path. The end-to-end path refers to: the business flow from the root cause abnormal GPU card (or called the root cause GPU card) flowing through the X-level networking to the GPU card of another business device, and the communication path formed by the network element devices passed in the X-level networking. In other words, only the root cause GPU card can restore the complete end-to-end path, and the non-root cause GPU card cannot restore the complete end-to-end path because the communication between the non-root cause GPU card and the root cause GPU card is interrupted. In other words, a complete end-to-end path means that a root cause GPU card in a service device in a service cluster and GPU cards in other service devices in the service cluster except the service device need to be connected through network element devices in the X-level networking, so that the root cause GPU card affects the task execution speed of the GPU card corresponding to the destination GPU card identifier through the X-level networking.

[0132] Further, it is mentioned in the aforementioned step S401 that the computer device can obtain the first communication element group of the full service flow within the abnormal time period in the service cluster; therefore, if there is an abnormal service flow in the service cluster, the first communication element group of the full service flow must include the second communication element group of the abnormal service flow reported by the TCCL module in the abnormal service device. The embodiment of the present application mainly analyzes the possible causes of abnormalities when the abnormal service flow is transmitted in the X-level networking when the service cluster is abnormal. Therefore, the computer device can compare the first communication element group and the second communication element group, specifically, according to the source GPU card identifier and the destination GPU card identifier in the abnormal second communication element group reported by the TCCL module, detect all first communication element groups with the same source GPU card identifier and destination GPU card identifier in the first communication element group of the full service flow; for ease of distinction, the first communication element group with the same source GPU card identifier and destination GPU card identifier can be called the third communication element group. If the first communication element group and the second communication element group are compared, the comparison result is: there are the first communication element group and the second communication element group with the same source GPU card identifier and destination GPU card identifier, indicating that the first communication element group with abnormal business flow is found from the first communication element group; that is, it is possible to eliminate the second communication element group that communicates through the hardware bus from the abnormal second communication element group, and the third communication element groups found are all communication element groups with abnormal business flows. If the first communication element group and the second communication element group are compared, the comparison result is empty, that is, there is no first communication element group and the third communication element group with the same source GPU card identifier and destination GPU card identifier, indicating that the cause of the abnormality in the business cluster may not be the abnormal business flow, and then exit the subsequent operations for the abnormal business flow.

[0133] For example, a schematic diagram of comparing the first communication element group and the second communication element group can be seen in Figure 5 .like Figure 5As shown, it is assumed that the business cluster includes business equipment A, business equipment B and business equipment C, each business equipment includes 8 GPU cards, the identifications of the 8 GPU cards included in business equipment A are A1, A2, A3, A4, A5, A6, A7 and A8 respectively, similarly, the identifications of the 8 GPU cards included in business equipment B are B1, B2, ..., B7 and B8 respectively, and the identifications of the 8 GPU cards included in business equipment C are C1, C2, ..., C7 and C8 respectively. Assume that the first communication element group of the business flow reported by the sflow agent to the computer device during the abnormal time period includes: (A1, B1, source port number 1, destination port number 1, protocol number), (B1, C2, source port number 1, destination port number 2, protocol number), (B3, C2, source port number 2, destination port number 3, protocol number); and the second communication element group reported by the abnormal business device includes: (B1, C2, source port number 1), (B1, B2, source port number 1), (B1, B3, source port number 1), (B1, B4, source port number 1) and (B1, B5, source port number 1), etc. Therefore, each second communication element group is compared with each first communication element group respectively, and the first communication element group whose source GPU card identifier and destination GPU card identifier are the same is found to be (B1, C2, source port number 1, destination port number 2, protocol number). Then the first communication element group (B1, C2, source port number 1, destination port number 2, protocol number) is used as the third communication element group. According to the third communication element group, the source and destination ends of the abnormal business flow can be determined, thereby determining the abnormal business flow.

[0134] In addition, from the perspective of the TCCL communication library implementation in the service device, a service flow corresponds to a queue pair (QueuePair, QP), and a queue pair consists of a send queue (Send Queue, SQ) and a receive queue (Receive Queue, RQ). The send queue is used to send messages, while the receive queue is used to process received messages. When the magSize (i.e., the size of the message, indicating the number of bytes of message data) of the service flow sent from the GPU card at the source end to the GPU card at the destination end exceeds a certain threshold (such as 32MB (megabytes)), the service flow will be sent to multiple queue pairs. At this time, the second communication element groups of each queue pair in the multiple queue pairs are the same. Therefore, there may be duplication in the second communication element group reported to the computer device by the TCCL module; for this purpose, after the computer device obtains one or more second communication element groups reported by abnormal services in the service cluster, it will first perform data deduplication processing on the one or more second communication element groups before comparing the second communication element groups with the first communication element groups. Specifically, deduplication is performed according to the source GPU card identifier, the destination GPU card identifier and the source port number, with the aim of deleting duplicate second communication element groups from the one or more second communication element groups to obtain deduplicated second communication element groups.

[0135] For example, assuming that the second communication element groups received by the computer device are: (B1, C2, source port number 1), (B1, C2, source port number 1), (B1, C2, source port number 1), (B1, B2, source port number 1), (B1, B3, source port number 1) and (B1, B4, source port number 1); when the computer device performs data deduplication processing on the second communication element groups, two of the three repeated second communication element groups (B1, C2, source port number 1), (B1, C2, source port number 1), (B1, C2, source port number 1) are deleted, and only one second communication element group is retained. Then, the second communication element groups after data deduplication processing include: (B1, C2, source port number 1), (B1, B2, source port number 1), (B1, B3, source port number 1) and (B1, B4, source port number 1).

[0136] Through the above-mentioned deduplication processing of the second communication element group, the duplicate second communication element group can be effectively removed. When comparing the second communication element group with the first communication element group, the resources and time cost required for the comparison can be reduced, and the accuracy of the comparison result can be improved.

[0137] It should be noted that the embodiments of the present application do not limit the order in which the computer device obtains the first communication element group and the second communication element group; for example, the computer device first obtains the first communication element group and then obtains the second communication element group, or the computer device first obtains the second communication element group and then obtains the first communication element group, or the computer device obtains the first communication element group and the second communication element group at the same time.

[0138] S404: Determine a communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow.

[0139] After obtaining the third communication element group of the abnormal business flow based on the above steps, the computer device can know the source and destination of the abnormal business flow according to the source GPU card identifier and the destination GPU card identifier in the third communication element group; and the X-level networking in the business cluster is determined, so the computer device can determine whether the communication path of the abnormal business flow exists based on the GPU card corresponding to the source end and the GPU card corresponding to the destination end in the third communication element group of the abnormal business flow, and the network element devices in the X-level networking. The communication path is an end-to-end path, and the communication path includes multiple target network element devices that the abnormal business flow passes through in sequence, that is, multiple target network element devices that the abnormal business flow passes through in sequence in the X-level networking. If the end-to-end path of the abnormal business flow exists, the abnormal business flow is the business flow corresponding to the root cause GPU card; if the end-to-end path of the abnormal business flow does not exist, the abnormal business flow is the business flow corresponding to the non-root cause GPU card affected by the root cause GPU card. Therefore, when the transmission path of the abnormal business flow in the X-level networking is in a black box state, the embodiment of the present application can restore the communication path of the abnormal business flow from the X-level networking in the black box state, which greatly facilitates the investigation of abnormal causes of the abnormal business flow and effectively ensures the accuracy of the investigation of abnormal causes.

[0140] The embodiment of the present application supports the use of a two-tuple to restore the communication path of the abnormal business flow. The two-tuple is the GPU card identifier corresponding to the source end and the GPU card identifier corresponding to the destination end in the five-tuple; that is, in the process of restoring the communication path of the abnormal business flow, the embodiment of the present application only uses the GPU card identifier corresponding to the source end and the GPU card identifier corresponding to the destination end in the five-tuple, which greatly simplifies the calculation. Figure 6a As shown, the computer device inputs the binary group of the abnormal business flow into the data preprocessing module, and uses the data preprocessing module to restore the communication path of the abnormal business flow according to the binary group of the abnormal business flow (i.e., the identification of the GPU card corresponding to the source end and the identification of the GPU card corresponding to the destination end) and the network element device in the X-level network. Among them, the data preprocessing module can be deployed in the computer device, or deployed in other devices independent of the computer device. The specific implementation process of the computer device calling the data preprocessing module to restore the communication path of the abnormal business flow can be seen in Figure 6b , including but not limited to steps s11-s13; wherein:

[0141] s11: The computer device obtains multiple path groups between X-layer subnetworks according to the GPU card corresponding to the source end of the abnormal business flow, the GPU card corresponding to the destination end of the abnormal business flow, and the network element devices in each layer of the subnetwork in the X-level networking.

[0142] When the computer device knows the source GPU card identifier corresponding to the source end of the abnormal business flow and the destination GPU card identifier corresponding to the destination end according to the third communication element group, the embodiment of the present application supports the use of the border gateway protocol (such as BGP protocol, see the above-mentioned related overview) between the X-level networking to restore the multiple path groups that the abnormal business flow may be transmitted between adjacent layer sub-networks. Among them, the path group is also called ECMP (Equal-Cost Multipath Routing, equal-cost path) group; each ECMP group includes one or more equal-cost paths from the same starting point to the same destination (or end point), and multiple equal-cost paths refer to multiple paths with the same path cost. The starting point and end point of each equal-cost path are network element devices in adjacent layer sub-networks, respectively. The starting point is called the starting network element device, and the end point is called the end network element device.

[0143] In a specific implementation, the specific implementation process of restoring the abnormal service flow by the computer device to multiple path groups that may be transmitted between adjacent layer subnetworks in the X-level networking may include but is not limited to:

[0144] (1) The computer device determines the starting network element device of the abnormal service flow in the first layer subnetwork in the X-level networking according to the identifier of the GPU card corresponding to the source end in the third communication element group of the abnormal service flow.

[0145] The computer device can determine that there is an abnormal source GPU card in the service cluster according to the identification of the GPU card corresponding to the source end in the third communication element group. Then, the connection link between the service device in the service device and the network element device in the first layer sub-network in the X-level networking is determined when the service cluster is constructed. Therefore, after determining the abnormal source GPU card, the computer device further determines the network element device connected to the abnormal source GPU card in the first layer sub-network in the X-level networking. At this time, the network element device connected to the abnormal source GPU card is called the starting network element device of the abnormal service flow.

[0146] (2) The computer device determines the terminal network element device of the abnormal service flow in the first layer subnetwork in the X-level networking according to the identifier of the GPU card corresponding to the destination end in the third communication element group of the abnormal service flow.

[0147] The computer device can determine that there is an abnormal destination GPU card in the service cluster based on the identifier of the GPU card corresponding to the destination in the third communication element group. Then, the connection link between the service device in the service device and the network element device in the first layer sub-network in the X-level networking is determined when the service cluster is constructed. Therefore, after determining the abnormal destination GPU card, the computer device further determines the network element device connected to the abnormal destination GPU card in the first layer sub-network in the X-level networking. At this time, the network element device connected to the abnormal source destination GPU card is called the terminal network element device of the abnormal service flow.

[0148] It should be noted that, in some cases, the computer device may not find the abnormal GPU card in the service cluster, or the starting network element device connected to the abnormal GPU card, based on the identifier of the GPU card corresponding to the source end in the third communication element group. Similarly, the computer device may not find the abnormal GPU card in the service cluster, or the terminal network element device connected to the abnormal GPU card, based on the identifier of the GPU card corresponding to the destination end in the third communication element group. For example, when equipment migration occurs in the service cluster, such as the migration of the starting network element device (such as equipment migration caused by replacing the optical module when the optical module in the network element device fails), the identifier of the abnormal GPU card will exist but the starting network element device cannot be found. When the computer device cannot find the starting network element device or the terminal network element device, it directly returns, that is, exits the subsequent restoration operation of the abnormal service flow.

[0149] (3) The computer device performs path connection processing in the X-layer subnetwork based on the starting network element device and the ending network element device to obtain multiple path groups between the X-layer subnetworks.

[0150] Specifically, in the process of searching for the path group between the subnetworks of the adjacent layers in the X-layer subnetwork based on the starting network element device and the ending network element device, if the computer device can find all the ECMP groups between the starting network element device and the ending network element device, indicating that the abnormal service flow has a complete end-to-end path, then it is considered that the starting network element device of the abnormal service flow and the first communication element group corresponding to the starting network element device are the communication element group of the root cause GPU card. On the contrary, if it is found that the path group between the subnetworks of any adjacent layers does not exist, such as one or more ECMP groups between the starting network element device and the ending network element device cannot be found, it is indicated that the abnormal service flow is a service flow affected by the root cause GPU card; because the communication has been interrupted at the root cause GPU card, the end-to-end path of the abnormal service flow cannot be restored in the X-level networking, then it is determined that the abnormality cannot be caused by the non-root cause GPU card of the abnormal service flow (that is, the GPU card corresponding to the source GPU card identifier and the GPU card corresponding to the destination GPU card identifier in the third communication element group of the abnormal service flow), then the abnormality analysis for the non-root cause GPU card is exited, and the abnormal cause analysis of the root cause GPU card continues.

[0151] The following is an introduction to the specific process of a computer device searching for a path group between subnetworks of adjacent layers in an X-layer subnetwork based on a starting network element device and an end network element device. For ease of explanation, in the embodiment of the present application, any layer of subnetwork in an X-layer subnetwork is represented as an i-th layer subnetwork, where i is an integer and 1≤i≤X; then the process of performing path connection processing in an X-layer subnetwork based on a starting network element device and an end network element device may include:

[0152] ① When i=1, according to the starting network element device and the i+1th layer sub-network, the border gateway protocol is used to query multiple first path groups corresponding to the starting network element device, and the opposite end device of each of the multiple first path groups is a network element device in the i+1th layer sub-network; in this case, the border gateway protocol defines: when i=1, starting from the starting network element device, one or more equivalent paths between the starting network element device and each network element device in the i+1th layer sub-network are found, and one or more equivalent paths between the starting network element device and any network element device in the i+1th layer sub-network are taken as a first path group, and the opposite end devices of the multiple first path groups are all network element devices in the i+1th layer sub-network. Among them, when i=1, the i+1th layer sub-network is the second layer sub-network in the X-level networking, and the opposite end device of each first path group is a network element device in the second layer sub-network (or expressed as the second layer sub-network); the so-called opposite end device of the first path group refers to the destination device reached by any path included in the first path group.

[0153] ② Let i=i+1, if i=X, then according to the peer device and the terminal network element device of each first path group, query the Xth path group corresponding to the peer device of each first path group according to the border gateway protocol, and the peer device of each first path group is the terminal network element device. In this case, the border gateway protocol defines: when i=X=2, starting from the peer device of each first path group, find one or more equivalent paths between the peer device of each first path group and the terminal network element device, and one or more equivalent paths between the peer device of a first path group and the terminal network element device are regarded as a first path group.

[0154] Since i=X=2, all possible transmission situations between X-level networks have been found, and it is determined that the multiple path groups between the X-level subnetworks include: multiple first path groups, and the X-th path group corresponding to the opposite end device of each first path group. For example, the X-level network in the service cluster is a secondary network. When i=i+1, i=2, and i=X=2 at this time, according to the opposite end device and the terminal network element device of the i-1th path group (the i-1th path group is the first path group, or expressed as the first path group), the second path group (or expressed as the second path group) corresponding to the opposite end device of each first path group is queried according to the border gateway protocol. In this case, the multiple paths between the second-layer subnetworks in the secondary network include: multiple first path groups and the second path group corresponding to the opposite end device of each first path group.

[0155] ③ Let i=i+1, if i<X, then according to the peer device of each i-1 path group in the multiple i-1 path groups and the i+1 layer sub-network, query the multiple i-th path groups corresponding to the peer device of each i-1 path group according to the border gateway protocol; the peer device of each i-th path group is the network element device in the i+1 layer sub-network. In this case, the border gateway protocol defines: when i<X, starting from the peer device of each i-1 path group, find one or more equivalent paths between the peer device of each i-1 path group and each network element device in the i+1 layer sub-network, and one or more equivalent paths between the peer device of an i-1 path group and a network element device in the i+1 layer sub-network as an i-th path group. For example, if X=3, that is, the X-level networking in the business cluster is a three-level networking, then when i=i+1, i=2, at this time i<X, then according to the opposite end device and the third layer subnetwork (or expressed as the third layer subnetwork) of each first path group in the multiple first path groups, the border gateway protocol is used to query the multiple second path groups corresponding to the opposite end device of each first path group.

[0156] ④ Loop through the previous step. When i=X, based on the peer device of each X-1th path group and the X-1th layer subnetwork, query the multiple X-th path groups corresponding to the peer device of each X-1th path according to the border gateway protocol. At this time, the peer device of any X-th path group is a network element device in the X-1th layer subnetwork. In this case, the border gateway protocol defines: when i=X and X is not equal to 2, starting from the peer device of each X-th path group, find one or more equivalent paths between the peer device of each X-th path group and each network element device in the X-1th layer subnetwork. For example, when X=3, based on the peer device of each second path group and the second layer subnetwork, query the multiple third path groups corresponding to the peer device of each second path group according to the border gateway protocol. At this time, the peer device of any third path group is a network element device in the second layer subnetwork.

[0157] ⑤ Let i=i-1, if i>2, then according to the peer device of each 2X-i-1 path group in the multiple 2X-i-1 path groups and the i-1 layer subnetwork, query the multiple 2X-i path groups corresponding to the peer device of each 2X-i-1 path group according to the border gateway protocol; the peer device of the 2X-i path group is a network element device in the i-1 layer subnetwork. In this case, the border gateway protocol defines: when i>2 and X is not equal to 2, starting from the peer device of each 2X-i-1 path group, find one or more equivalent paths between the peer device of each 2X-i-1 path group and each network element device in the i-1 layer subnetwork. For example, when X=5 and i=4, according to the peer device of each fifth path group in the multiple fifth path groups and the third layer subnetwork, find the multiple sixth path groups corresponding to the peer device of each fifth path group according to the border gateway protocol.

[0158] ⑥ Loop through the previous step. When i=2, query the 2X-2 path group corresponding to the peer device of each 2X-i-1 path group according to the border gateway protocol based on the peer device and the terminal network element device of each 2X-i-1 path group; the peer device of the 2X-2 path group is the terminal network element device. In this case, the border gateway protocol defines: when i=2 and X is not equal to 2, start from the peer device of each 2X-i-1 path group, and find one or more equivalent paths from the peer device of each 2X-i-1 path group to the terminal network element device.

[0159] At the end of step ⑥, i=2, then all possible transmission situations between the X-level networks have been found, and the multiple path groups between the X-layer subnetworks are determined to include: multiple first path groups, multiple i-th path groups, multiple 2X-i path groups, and 2X-2 path groups corresponding to the opposite devices of the 2X-i path groups.

[0160] It should be noted that the above steps ① to ⑥ summarize the specific implementation process of restoring or determining multiple path groups of abnormal business flows between X-layer sub-networks in each networking case where X≥2 according to the identification of the GPU card corresponding to the source end and the identification of the GPU card corresponding to the destination end in the third communication element group. Among them, steps ① and ② are the specific implementation process of restoring multiple path groups of abnormal business flows between the second-layer sub-networks when X=2, that is, when the X-level networking included in the business cluster is a second-level networking; steps ①, ③-⑥ are the specific implementation process of restoring multiple path groups of abnormal business flows between the X-layer sub-networks when X>2, that is, when the number of levels of the X-level networking included in the business cluster is greater than 2 (such as a third-level networking). In addition, if in any of the above steps ① to ⑥ or steps ①, ③ to ⑥, the path group is not found based on the third communication element group (for example, due to the influence of the root cause GPU card, the communication is interrupted at the root cause GPU card, resulting in the non-root cause GPU card being unable to find the path group in the X-level network), then the third communication element group is determined to be the communication element group of the non-root cause GPU card, and the process is exited.

[0161] For ease of understanding, the specific implementation process of restoring multiple path groups of abnormal service flows between X-layer subnetworks when X=2 and X=3 is exemplarily introduced below with reference to the accompanying drawings.

[0162] When X=2, the flowchart of restoring multiple path groups of abnormal business flows between Layer 2 sub-networks can be seen in Figure 7 .like Figure 7 As shown, it is assumed that the GPU card corresponding to the source end in the third communication element group is GPU card A1 in service device A, and the GPU card corresponding to the destination end is GPU card B2 in service device B; the first layer sub-network in the secondary network includes three network element devices, namely CPULA1, CPULA2 and CPULA3, and the second layer sub-network includes three network element devices, namely CPULC1, CPULC2 and CPULC3; and the starting network element device is CPULA1 in the first layer sub-network, and the end network element device is CPULA2 in the first layer sub-network. Then, based on Figure 7 The process of restoring multiple path groups between the second-layer subnetworks in the secondary networking shown includes: when i=1, according to the starting network element device CPULA1 and the second-layer subnetwork, querying multiple first path groups corresponding to the starting network element device CPULA1 according to the border gateway protocol; wherein the number of first path groups is the same as the number of network element devices in the second-layer subnetwork, and the three first path groups are: GPULA1→GPULC1, GPULA1→GPULC2, GPULA1→GPULC3, and the number of equivalent paths in each first path group can be one or more, which is not limited.

[0163] Further, let i=i+1=2, and then determine i=X. Then, according to the peer device of each first path group and the terminal network element device CPULA2, query the second path group corresponding to the peer device of each first path group according to the border gateway protocol. Among them, the peer device of the first path group GPULA1→GPULC1 is GPULC1, and the second path group corresponding to the peer device GPULC1 is GPULC1→CPULA2; similarly, the peer device of the first path group GPULA1→GPULC2 is GPULC2, and the second path group corresponding to the peer device GPULC2 is GPULC2→CPULA2; similarly, the peer device of the first path group GPULA1→GPULC3 is GPULC3, and the second path group corresponding to the peer device GPULC3 is GPULC3→CPULA2. In summary, the computer device obtains three first path groups and three second path groups under the secondary networking.

[0164] When X=3, the flowchart of restoring multiple path groups of abnormal business flows between three-layer sub-networks can be seen in Figure 8 .like Figure 8 As shown, it is assumed that the GPU card corresponding to the source end in the third communication element group is GPU card A1 in service device A, and the GPU card corresponding to the destination end is GPU card B2 in service device B; the first layer sub-network in the three-level network includes three network element devices, namely CPULA1, CPULA2 and CPULA3, the second layer sub-network includes three network element devices, namely CPULC1, CPULC2 and CPULC3, and the third layer sub-network includes two network element devices, namely SGLC1 and SGLC2; and the starting network element device is CPULA1 in the first layer sub-network, and the end network element device is CPULA2 in the first layer sub-network. Then, based on Figure 8 The process of restoring multiple path groups between three-layer subnetworks in the three-level networking shown includes: when i=1, according to the starting network element device CPULA1 and the second-layer subnetwork, query the multiple first path groups corresponding to the starting network element device CPULA1 according to the border gateway protocol; wherein the number of first path groups is the same as the number of network element devices in the second-layer subnetwork, and the three first path groups are: GPULA1→GPULC1, GPULA1→GPULC2, GPULA1→GPULC3, and the number of equivalent paths in each first path group can be one or more, which is not limited.

[0165] Further, let i=i+1=2, and at this time determine that i<X, then according to the peer device and the third layer subnetwork of each first path group in the multiple first path groups, query the multiple second path groups corresponding to the peer device of each first path group according to the border gateway protocol. Among them, the peer device of the first path group GPULA1→GPULC1 is GPULC1, and the second path groups corresponding to the peer device GPULC1 are GPULC1→SGLC1 and GPULC1→SGLC2; similarly, the peer device of the first path group GPULA1→GPULC2 is GPULC2, and the second path groups corresponding to the peer device GPULC2 are GPULC2→SGLC1 and GPULC2→SGLC2; similarly, the peer device of the first path group GPULA1→GPULC3 is GPULC3, and the second path groups corresponding to the peer device GPULC3 are GPULC3→SGLC1 and GPULC3→SGLC2.

[0166] Furthermore, let i=i+1=3, and then determine i=X. Then, according to the peer device of each X-1th path group (i.e., the second path group) and the X-1th layer subnetwork (i.e., the second layer subnetwork), query the multiple Xth path groups corresponding to the peer devices of each X-1th path group according to the border gateway protocol. Among them, the peer devices of the second path group GPULC1→SGLC1, the second path group GPULC2→SGLC1, and the second path group GPULC3→SGLC1 are all SGLC1, and the third path groups corresponding to the peer device SGLC1 are: SGLC1→GPULC1, SGLC1→GPULC2, and SGLC1→GPULC3; similarly, the peer devices of the second path group GPULC1→SGLC2, the second path group GPULC2→SGLC2, and the second path group GPULC1→SGLC2 are all SGLC2, and the third path groups corresponding to the peer device SGLC2 are SGLC2→GPULC1, SGLC2→GPULC2, and SGLC2→GPULC3.

[0167] Furthermore, let i=i-1=2. At this time, it is determined that i=2. Then, according to the opposite end device of each 2X-i-1 path group (i.e., the third path group) and the terminal network element device GPULA2, the 2X-2 path group corresponding to the opposite end device of each 2X-i-1 path group is queried according to the border gateway protocol. Among them, the opposite end devices of the third path group SGLC1→GPULC1 and the third path group SGLC2→GPULC1 are both GPULC1, and the fourth path group corresponding to the opposite end device GPULC1 is GPULC1→GPULA2; similarly, the opposite end devices of the third path group SGLC1→GPULC2 and the third path group SGLC2→GPULC2 are both GPULC2, and the fourth path group corresponding to the opposite end device GPULC2 is GPULC2→GPULA2; similarly, the opposite end devices of the third path group SGLC1→GPULC3 and the third path group SGLC2→GPULC3 are both GPULC3, and the fourth path group corresponding to the opposite end device GPULC3 is GPULC3→GPULA2. In summary, the computer device obtains three first path groups, six second path groups, six third path groups and three fourth path groups under the secondary networking.

[0168] s12: The computer device constructs a directed graph according to the multiple path groups and the flow direction of the abnormal service flow between the X-layer sub-networks.

[0169] A directed graph is a directional graph, which consists of a set of nodes and directed edges, and each directed edge connects a pair of ordered nodes. As mentioned above, the abnormal service flow from the starting network element device to the end network element device in the X-level networking is also directional, so the embodiment of the present application supports constructing a directed graph corresponding to the abnormal service flow according to multiple path groups between the X-layer subnetworks found in the X-level networking in step s11.

[0170] In the specific implementation, the computer device uses multiple network element devices in multiple path groups between the X-layer subnetworks as nodes of a directed graph; then, according to the flow direction of the abnormal business flow between the X-layer subnetworks, multiple nodes are connected to obtain a directed graph corresponding to the abnormal business flow. The directed graph includes a start node, an end node, and X-1 intermediate layers, each of which includes one or more intermediate nodes. Take the X-level networking as a secondary networking as an example, Fig. 9 As shown, according to Figure 7The nodes in the directed graph constructed by the multiple path groups shown include: the starting node GPULA1, the intermediate nodes GPULC1, GPULC2 and GPULC3, and the terminal node GPULA2. Among them, the starting node GPULA1 is connected to the intermediate nodes GPULC1, GPULC2 and GPULC3 respectively, and the arrow directions after the connection are GPULA1→GPULC1, GPULA1→GPULC and GPULA1→GPULC3, and each intermediate node is connected to the terminal node GPULA2, and the arrow directions after the connection are GPULC1→GPULA1, GPULC2→GPULA1 and GPULC3→GPULA1.

[0171] s13: The computer device performs path search processing on the directed graph to obtain the communication path of the abnormal business flow.

[0172] Among them, the communication path of the abnormal business flow can be understood as the end-to-end path corresponding to the root cause GPU card. The directed graph corresponding to the abnormal business flow is a weighted directed graph; the so-called weighted directed graph means that each edge in the directed graph has a weight value, and the weight value of any edge can represent the distance or time and other parameters between one node of the edge and another node. In the subsequent embodiments, the weight value of the edge represents the distance between two nodes as an example for explanation. Then the larger the weight value of the edge, the larger the distance between the two nodes of the edge, and vice versa. The smaller the weight value of the edge, the smaller the distance between the two nodes of the edge. The embodiment of the present application supports path search processing for weighted directed graphs, aiming to search out the shortest path from the directed graph as the communication path of the abnormal business flow; this is to take into account that the border gateway protocol between hierarchical subnetworks also supports the selection of the shortest path to transmit business flows, so the searched shortest path is used as the communication path of the abnormal business flow. It is more in line with the transmission of business flows between X-level networks in real situations.

[0173] The embodiment of the present application supports the use of a path traversal algorithm to search for the shortest path from a weighted directed graph; for example, the path traversal algorithm is a depth first search (DFS) algorithm. The following takes any one of the X-1 intermediate layers of the directed graph as the jth intermediate layer as an example to introduce the process of searching for the shortest path from the directed graph, where j is an integer and 1≤j≤X-1.

[0174] In a specific implementation, when j=1, starting from the starting node of the directed graph, the path between the starting node and each intermediate node in the first intermediate layer of the X-1 intermediate layers is traversed to obtain multiple first weight values ​​corresponding to the starting node; one first weight value is the weight value of the path between the starting node and an intermediate node in the first intermediate layer, such as the weight value is the distance. Then, let j=j+1, when j>X-1, the Xth weight value of the path between each intermediate node in the j-1th intermediate layer and the terminal node is calculated respectively; an Xth weight value is the weight value of the path between an intermediate node in the j-1th intermediate layer and the point node. In this case, the computer device determines the shortest path from the directed graph as the communication path of the abnormal business flow based on the multiple first weight values ​​and the Xth weight value. Conversely, if j≤X-1, starting from each intermediate node in the j-1th intermediate layer, the path between each intermediate node in the j-1th intermediate layer and each intermediate node in the j-1th intermediate layer is traversed to obtain multiple jth weight values ​​corresponding to each intermediate node in the j-th intermediate layer. In this case, the computer device determines the shortest path as the communication path of the abnormal business flow from the directed graph based on multiple first weight values, multiple j-th weight values ​​corresponding to each intermediate node in the j-th intermediate layer, and the X-th weight value.

[0175] Taking X=2 as an example, the exemplary process of determining the shortest path from a directed graph can be found in Fig.10a .like Fig.10a As shown, when j=1, the three first weight values ​​corresponding to the starting node are calculated, that is, the weight values ​​of the edges from the starting node GPULA1 to the intermediate nodes GPULC1, GPULC2 and GPULC3 are 1, 3 and 2 respectively; when j=j+1=2, it is determined that j>X-1=1, and the Xth weight value (i.e., the second weight value) of the path between each intermediate node in the first intermediate layer and the terminal node GPULA2 is calculated respectively, such as the second weight values ​​of the edges from the intermediate node GPULC1, the intermediate node GPULC2 and the intermediate node GPULC3 to the terminal node are 2, 1 and 2 respectively. Then, based on the three first weight values ​​and the three second weight values, the shortest path is determined from the directed graph as GPULA1→GPULC1→GPULLA2, and the end-to-end path GPULA1→GPULC1→GPULLA2 is used as the communication path of the abnormal service flow.

[0176] Taking X=3 as an example, the exemplary process of determining the shortest path from a directed graph can be found in Fig.10b .like Fig.10bAs shown, when j=1, the three first weight values ​​corresponding to the starting node are calculated, that is, the weight values ​​of the edges from the starting node GPULA1 to the intermediate nodes GPULC1, GPULC2 and GPULC3 are 1, 3 and 2 respectively. Let j=j+1=2, at this time j=X-1=2, the computer device starts from each intermediate node in the first intermediate layer, traverses the path between each intermediate node in the second intermediate layer, and obtains multiple second weight values ​​corresponding to each intermediate node in the second intermediate layer. Exemplarily, the two second weight values ​​of the edges from the intermediate node GPULC1 in the first intermediate layer to the intermediate nodes SGLC1 and SGLC2 in the second intermediate layer are 1 and 1 respectively; similarly, the two second weight values ​​of the edges from the intermediate node GPULC2 in the first intermediate layer to the intermediate nodes SGLC1 and SGLC2 in the second intermediate layer are 1 and 2 respectively; similarly, the two second weight values ​​of the edges from the intermediate node GPULC3 in the first intermediate layer to the intermediate nodes SGLC1 and SGLC2 in the second intermediate layer are 2 and 3 respectively. Further, let j=j+1=3, at this time j>X-1=3, then calculate the Xth weight value of the path between each intermediate node and the terminal node in the second intermediate layer respectively; wherein, the third weight value of the edge between the intermediate node SGLC1 in the second intermediate layer and the terminal node GPULA2 is 2, and the third weight value of the edge between the intermediate node SGLC2 in the second intermediate layer and the terminal node GPULA2 is 1. Finally, based on the three first weight values, the three second weight values ​​and the two third weight values, the shortest path is determined to be GPULA1→GPULC1→SGLC2→GPULA2 from the directed graph, and the end-to-end path GPULA1→GPULC1→SGLC2→GPULA2 is used as the communication path of the abnormal service flow.

[0177] It should be noted that the above steps s11-s13 take the X-level networking in the service cluster as a secondary networking and a tertiary networking as an example, and introduce the specific implementation process of the computer device restoring the communication path of the abnormal service flow in the X-level networking according to the border gateway protocol, and will not limit the embodiment of the present application. In other words, the above-mentioned process of restoring the communication path provided in the embodiment of the present application can also be applied to a more complex X-level networking architecture, which is specially explained here.

[0178] In summary, through the specific implementation process shown in the above steps S401-S404, the computer device can construct a directed graph corresponding to the GPU card corresponding to the source end of the abnormal business flow and the GPU card corresponding to the destination end based on the first communication element group of the full business flow in the business cluster and the second communication element group reported by the abnormal business device, and then search the shortest path from the directed graph as the communication path of the abnormal business flow. The above process does not require human participation, greatly speeds up the restoration efficiency of the communication path of the abnormal business flow, thereby reducing the time to find the fault node, and can achieve the fault discovery time from an average of 4 hours to less than 3 minutes, and improve the troubleshooting efficiency by 80 times.

[0179] To facilitate a coherent understanding of the specific implementation process shown in steps S401-S404, Fig.11 Taking the X-level networking as the second-level networking and the third-level networking, and the border gateway protocol as the BGP neighbor routing as an example, the overall process of the computer equipment quickly recovering the communication path of the abnormal business flow is introduced. Fig.11 As shown: the computer device obtains the first communication element group of the full service flow in the service cluster and the second communication element group reported by the abnormal service device. The computer device compares the first communication element group with the second communication element group. If the comparison result is: there is no first communication element group and second communication element group with the same source GPU card identifier and destination GPU card identifier, then the process is exited; if the comparison result is: there is a first communication element group and second communication element group with the same source GPU card identifier and destination GPU card identifier, then the first communication element group (or third communication element group) of the abnormal service flow is obtained. The computer device detects whether there is a starting network element device in the service cluster according to the GPU card identifier corresponding to the source end in the first communication element group of the abnormal service flow (i.e., the source GPU card identifier), and detects whether there is a terminal network element device in the service cluster according to the GPU card identifier corresponding to the destination end in the first communication element group of the abnormal service flow (i.e., the destination GPU card identifier); if any of the starting network element device and the terminal network element device does not exist, the process is exited; if both the starting network element device and the terminal network element device exist, the computer device finds a first path group starting from the starting network element device according to the next-hop BGP neighbor routing of the starting network element device, and the opposite end device of the first path group is a GPULC device in the second layer subnetwork in the X-level networking; and then finds a second path group starting from the opposite end device of the first path group according to the next-hop BGP neighbor routing of the opposite end device of the first path group.

[0180] When the X-level networking is a two-level networking, the opposite device of the second path group is a GPULA device in the first layer sub-network of the X-level networking, specifically, a terminal network element device in the first layer sub-network. At this time, a directed graph corresponding to the abnormal service flow is constructed according to multiple first path groups and multiple second path groups. When the X-level networking is a three-level networking, the opposite device of the second path group is an SGLC device in the third layer sub-network of the three-level networking, then the computer device continues to find the third path group starting from the opposite device of the second path group according to the next hop BGP neighbor route of the opposite device of the second path group, and the opposite device of the third path group is a GPULC device; the computer device continues to find the fourth path group starting from the opposite device of the third path group according to the next BGP neighbor route of the opposite device of the third path group, and the opposite device of the fourth path group is a GPULA device in the first layer sub-network of the X-level networking, specifically, a terminal network element device in the first layer sub-network. At this time, a directed graph corresponding to the abnormal service flow is constructed according to multiple first path groups, multiple second path groups, multiple third path groups, and multiple fourth path groups. Finally, after constructing the directed graph of the abnormal business flow, the computer device uses a path traversal algorithm to search for the shortest path from the weighted directed graph as the end-to-end path of the abnormal business flow.

[0181] S405: Perform device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow.

[0182] After restoring the end-to-end path of the abnormal business flow from the X-level networking of the black box based on the aforementioned steps, the embodiment of the present application supports device analysis and processing of the target network element device on the end-to-end path of the abnormal business flow according to the basic detection module, aiming to analyze whether the abnormal business flow is caused by the failure of the target network element device or the abnormal business flow caused by the network failure of the X-level networking, thereby realizing the abnormal cause analysis of the abnormal business flow, facilitating the operation and maintenance personnel to better adjust the business cluster based on the abnormal cause obtained by the analysis, thereby realizing the rapid recovery and execution of the task. Among them, the basic detection module can be a module deployed in a computer device, or deployed in other devices independent of the computer device; the basic detection module has the ability to perform device analysis and processing on the target network element device in the end-to-end path of the abnormal business flow.

[0183] In summary, on the one hand, the embodiment of the present application effectively restores the communication path of the root cause GPU card when there is an abnormality in the business cluster. Compared with analyzing all business devices in the business cluster when the fault occurs, the scope of the troubleshooting that needs to be investigated is narrowed from all business devices participating in the task in the business cluster to abnormal business devices that only have a communication relationship and upload the second communication element group during the abnormal time period, specifically the communication path of the root cause GPU card; it greatly improves the processing efficiency of faults in the business cluster, avoids the waste of computing power of business devices in the business cluster, and saves the cost of task execution. On the other hand, for network operation and maintenance, many faults manifest as suspected network problems during the task execution. Through the processing solution of the business cluster provided by the embodiment of the present application, it can be accurately determined whether it is a network problem or a problem on the terminal side (i.e., the target network element device), and the accurate positioning of fault troubleshooting can be achieved, thereby helping the rapid recovery of business execution during the task execution process.

[0184] Fig.12 A flowchart of another service cluster processing method provided by an exemplary embodiment of the present application is shown. The processing method can be executed by a computer device, which is the aforementioned Figure 3 The computer device 308 shown is, for example, a terminal or a server. The processing method of the service cluster may include but is not limited to steps S1201-S1208:

[0185] S1201: Obtain a first communication element group of multiple service flows in a service cluster.

[0186] S1202: Acquire a second communication element group of abnormal service devices in the service cluster.

[0187] S1203: Filter out abnormal service flows from the service cluster based on the first communication element group and the second communication element group.

[0188] S1204: Determine a communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow.

[0189] It should be noted that the specific implementation process shown in steps S1201-S1204 is the same as the above Figure 4 The specific implementation process shown in steps S401-S404 in the illustrated embodiment is the same, and reference may be made to the relevant description of the specific implementation process shown in steps S401-S404, which will not be repeated here.

[0190] In addition, the embodiment of the present application supports visual display of the results of the path restoration performed by the data preprocessing module for the first communication element group and the second communication element group, so that the operation and maintenance personnel can intuitively perceive the restoration of the real transmission path for the third communication element group. For example, after a directed graph is constructed based on the first communication element group and the second communication element group, the visualization display of the end-to-end path of the abnormal service flow found in the directed graph can be seen in Fig.13a ;like Fig.13a As shown, a third communication element group and an end-to-end path restored based on the third communication element group are displayed in a computer device (such as a terminal with a display screen). For example, a schematic diagram of an end-to-end path of an abnormal service flow that is not restored based on the first communication element group and the second communication element group can be found in Fig.13b ;like Fig.13b As shown, the third communication element group is displayed in the computer device, and the result prompt information 1301 is displayed. The result prompt information 1301 is used to prompt that the end-to-end path corresponding to the third communication element group is not found, that is, the end-to-end path cannot be restored according to the third communication element group. At this time, the abnormal business flow corresponding to the third communication element group is the business flow affected by the root cause GP card, rather than the business flow corresponding to the root cause GPU card.

[0191] S1205: Acquire a reference path, and compare multiple reference network element devices on the reference path with multiple target network element devices on the communication path to obtain a comparison result.

[0192] S1206: If the comparison result is: the multiple reference network element devices are different from the multiple target network element devices, or the communication connection sequence between the multiple reference network element devices is different from the communication connection sequence between the multiple target network element devices, then the abnormal cause of the chip is analyzed to obtain abnormal information of the chip.

[0193] S1207: If the comparison result is: multiple reference network element devices are the same as multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, then trigger the step of performing device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal business flow.

[0194] In steps S1205-S1207, in an embodiment of the present application, after the computer device restores the end-to-end path of the abnormal business flow based on the aforementioned steps, it also supports further rationality analysis of the parallel communication path adopted by the operation and maintenance personnel in the business cluster according to the restored real end-to-end path (or called the real communication path), to help the operation and maintenance personnel to reasonably plan the hardware deployment of the business cluster. Among them, a chip is deployed on the network element device in the business cluster, and the chip can be a network interface board or a network interface card, or a network card, which is used to plan or specify the transmission path of the business flow that needs to be transmitted through the network element device according to the first communication element group of the business flow collected by the sflow agent. Among them, the rationality analysis of the parallel communication path mainly refers to the rationality analysis of the path planning capability of the chip deployed by the operation and maintenance personnel in the network element device to detect whether the ability of the chip to plan the path achieves the expected effect. If the expected effect is not achieved, the operation and maintenance personnel can replace the chip to improve the ability of the chip in the network element device in the business cluster to plan the transmission path.

[0195] Taking into account that the chip in the network element device itself has a fault, or the chip's ability to plan the transmission path is weak, it will cause abnormal task execution in the service cluster; based on this, the embodiment of the present application supports a rationality analysis of the chip before performing device analysis and processing on the target network element device on the end-to-end path of the abnormal service flow, to check whether the task execution abnormality is caused by the chip, thereby realizing more effective and accurate troubleshooting.

[0196] In a specific implementation, a computer device obtains a reference path planned by a chip based on the first communication element group of an abnormal business flow, and the reference path includes: a business flow between a GPU card corresponding to the source end of the abnormal business flow and a GPU card corresponding to the destination end of the abnormal business flow, and multiple reference network element devices passed through in the X-level networking. In other words, the reference path is a path planned for the abnormal business flow by the chip in the network element device according to the first communication element group after receiving the first communication element group of the abnormal business flow collected by the sflow agent. Further, the computer device compares the reference path planned by the chip with the real communication path of the abnormal business flow restored according to the embodiment of the present application, specifically comparing multiple reference network element devices on the reference path with multiple target network element devices on the real communication path to obtain a comparison result; that is, the comparison here not only compares whether the reference network element device and the target network element device are the same, but also compares whether the communication connection sequence between multiple reference network element devices and the communication connection sequence between multiple target network element devices are the same.

[0197] If the comparison result is: the number of reference network element devices is the same as the number of target network element devices, each of the multiple reference network element devices is the same as one of the multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, indicating that the reference path planned or specified by the chip based on the first communication element group of the abnormal business flow is the same as the actual communication path restored according to steps S1201-S1204. It is determined that the chip's ability to plan the transmission path achieves the expected effect; in this case, execute the following step S1208 to analyze the target network element devices on the communication path of the abnormal business flow to analyze the cause of the abnormal business flow.

[0198] If the comparison result is: the number of reference network element devices is different from the number of target network element devices, or any reference network element device among the multiple reference network element devices is different from one of the multiple target network element devices, or the communication connection order between the multiple reference network element devices is different from the communication connection order between the multiple target network element devices, it indicates that the reference path planned or specified by the chip based on the first communication element group of the abnormal business flow is different from the actual communication path restored according to steps S1201-S1204. It is determined that the chip's ability to plan the transmission path does not achieve the expected effect, and the generation of the abnormal business flow may be caused by the chip; in this case, the operation and maintenance personnel need to first analyze the cause of the chip abnormality and obtain the chip's abnormal information. The chip's abnormal information describes the cause of the chip abnormality, so that the chip's abnormal information can be used to guide the operation and maintenance personnel to adjust and optimize the chip.

[0199] S1208: Perform device analysis on each target network element device on the communication path to obtain abnormal information of the abnormal service flow.

[0200] Device analysis and processing is performed on each target network element device on the communication path of the abnormal service flow, aiming to analyze whether the abnormal service flow is caused by the network or the target network element device. In an embodiment of the present application, it supports the comparison of network indicators of each target network element device on the communication path to realize device analysis and processing. Specifically, the computer device calls the basic detection module to compare the network indicators of each target network element device on the communication path; wherein, the process of the basic detection module comparing the network indicators of each target network element device on the communication path may include: obtaining one or more detection indicators, and the detection indicators include at least one of the following: traffic congestion indicator, congestion level indicator, GPU utilization and RDMA traffic, etc. Then, each detection indicator among one or more detection indicators is used to perform indicator analysis and processing on each target network element device among multiple target network element devices, and the indicator information of the target network element device under each detection indicator is obtained. For example, if the number of detection indicators is 3, then each target network element device corresponds to 3 indicator information, and one indicator information corresponds to one detection indicator; the indicator information of the target network element device under any detection indicator is the indicator data obtained by the basic detection module using any detection indicator to perform indicator analysis and processing on the target network element device after the target network element device is input into the basic detection module. In this way, according to the indicator information of each target network element device under each detection indicator, an abnormal analysis is performed on the abnormal business flow to obtain the abnormal information of the abnormal business flow; specifically, the indicator information of the same target network element device under each detection indicator is integrated, and the target network element device is comprehensively analyzed to obtain the device analysis result of each target network element device after the comprehensive analysis, and then the abnormal business flow is abnormally analyzed according to the device analysis result of each target network element device to obtain the abnormal information of the abnormal business flow.

[0201] For example, the result diagram of using PFC traffic and ECN messages to perform indicator analysis on target network element devices on the end-to-end path of abnormal service flows can be seen in Fig.14 ;like Fig.14 As shown, assuming that the end-to-end path is: GPULA-006→GPULC-0614→GPULA-014. Then, the computer device inputs the end-to-end path GPULA-006→GPULC-0614→GPULA-014 into the basic detection module. The basic detection module uses two detection indicators, PFC flow and ECN message, to perform indicator analysis and processing on each target network element device in the end-to-end path, and obtains the comprehensive indicator information 1401 corresponding to the target network element device GPULA-006, the comprehensive indicator information 1402 corresponding to the target network element device GPULC-0614, and the comprehensive indicator information 1403 corresponding to the target network element device GPULA-014. Fig.14As shown, the indicator information of each target network element device under each detection indicator can be displayed in graphics and text; for example, the comprehensive indicator information of the target network element device GPULA-006 includes both indicator information 14011 corresponding to the detection indicator "PFC flow" and indicator information 14012 corresponding to the detection indicator "ECN message". Fig.14 In the indicator information shown, the indicator information of the three target network element devices under the ECN indicator is all 0. Then, looking at the indicator information of the three target network element devices under the PFC indicator, the indicator information of the three target network element devices under the PFC indicator is not 0. The indicator information of the three target network element devices under the PFC indicator is accumulated and calculated for a period of time before and after the fault. Analysis shows that a large number of PFC messages appear on each target network element device through which the abnormal service flow passes, thereby determining that the abnormality is caused by a network problem of the X-level networking, rather than an abnormality caused by a problem with the target network element device itself (optical module failure or excessive temperature, etc.).

[0202] It can be seen that the method of using graphic and text forms to comprehensively display the indicator information of the target network element equipment under various detection indicators greatly facilitates the operation and maintenance personnel to analyze the target network element equipment by combining multiple detection indicators at the same time, and facilitates the operation and maintenance personnel to analyze the abnormal causes of abnormal business flows faster and more accurately.

[0203] Further, the embodiment of the present application supports the computer device to perform cluster adjustment processing on the service cluster according to the abnormal information of the abnormal service flow to obtain the adjusted service cluster; the service cluster is adjusted to ensure that the adjusted service cluster will not generate abnormal service flows due to the same abnormal reason when executing tasks. For example, if the abnormal reason described by the abnormal information of the abnormal service flow is: a target network element device on the end-to-end path of the abnormal service flow fails, then a new network element device without a fault is used to replace the target network element device to ensure the rapid recovery and execution of the service. For another example, if the abnormal reason described by the abnormal information of the abnormal service flow is: the service execution is abnormal due to network problems, the network problem is further investigated, such as restarting the network. For another example, if the abnormal reason described by the abnormal information of the abnormal service flow is: the GPU card corresponding to the source end of the abnormal service flow fails, then the GPU card corresponding to the source end of the abnormal service flow is replaced in the service cluster to obtain a replaced service cluster, and the replaced service cluster no longer contains the faulty GPU card, thereby realizing the rapid recovery and execution of the service. It should be noted that, depending on the different abnormal causes described by the abnormal information of the abnormal business flow, the operation and maintenance personnel will have different ways of adjusting the cluster of the business cluster; the above only gives three exemplary methods of adjusting the cluster of the business cluster under three exemplary abnormal causes, which will not limit the embodiments of the present application, and is specially explained here.

[0204] In summary, the processing solution for the service cluster provided in the embodiment of the present application can quickly discover the cause of the fault in the service cluster, quickly locate the end-to-end path of the fault, effectively reduce the troubleshooting time, and thus improve the service recovery efficiency. On the one hand, it can reduce the fault discovery time and significantly improve the troubleshooting efficiency. On the other hand, for network operation and maintenance, the service cluster troubleshooting is no longer vaguely positioned as a suspected network problem, but it can accurately determine whether the fault is a network problem or an end-side (network element device) problem, thereby improving the accuracy of troubleshooting and achieving rapid recovery of service execution.

[0205] The above detailed description of the method of the embodiment of the present application, in order to facilitate the above-mentioned scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that contains the function of the module or unit.

[0206] Fig.15 A schematic diagram of the structure of a processing device of a service cluster provided by an exemplary embodiment of the present application is shown; the service cluster includes multiple service devices and multiple network element devices, and any two service devices are connected to each other through at least one network element device; each service device includes at least one GPU card, and the processing device can be used to execute Figure 4 and Fig.12 Some or all of the steps in the method embodiment shown. Fig.15 , the device comprises the following units:

[0207] The acquisition unit 1501 is used to acquire a first communication element group of multiple service flows in a service cluster, where the service flow refers to a data flow from a GPU card in a service device at a source end to a GPU card in a service device at a destination end; the first communication element group includes an identifier of a GPU card corresponding to a source end of the service flow and an identifier of a GPU card corresponding to a destination end;

[0208] The acquisition unit 1501 is further used to acquire a second communication element group of an abnormal service device in the service cluster, where the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card;

[0209] The processing unit 1502 is configured to filter out abnormal service flows from the service cluster based on the first communication element group and the second communication element group;

[0210] The processing unit 1502 is further configured to determine a communication path of the abnormal service flow according to a GPU card corresponding to a source end and a GPU card corresponding to a destination end of the abnormal service flow, wherein the communication path includes a plurality of target network element devices that the abnormal service flow passes through in sequence;

[0211] The processing unit 1502 is further configured to perform device analysis on each target network element device on the communication path to obtain abnormal information of the abnormal service flow.

[0212] In one implementation, the service cluster includes an X-level network, the X-level network includes a hierarchically distributed X-layer subnetwork, each layer of the subnetwork includes a plurality of network element devices, X is an integer and X≥2; the processing unit 1502 is used to determine the communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, specifically for:

[0213] According to the GPU card corresponding to the source end of the abnormal business flow, the GPU card corresponding to the destination end of the abnormal business flow, and the network element devices in each layer of the sub-network in the X-level networking, multiple path groups between the X-layer sub-networks are obtained; each path group includes one or more equal-cost paths, and the two path endpoints of the equal-cost path are respectively network element devices in two adjacent layers of sub-networks;

[0214] According to the flow directions of multiple path groups and abnormal business flows between the X-layer sub-networks, a directed graph is constructed;

[0215] Perform path search processing on the directed graph to obtain the communication path of the abnormal business flow.

[0216] In one implementation, the processing unit 1502 is used to obtain multiple path groups between X-layer subnetworks according to the GPU card corresponding to the source end of the abnormal service flow, the GPU card corresponding to the destination end of the abnormal service flow, and the network element device in each layer of the subnetwork in the X-level networking, specifically to:

[0217] According to the identification of the GPU card corresponding to the source end of the abnormal service flow, the starting network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0218] According to the identification of the GPU card corresponding to the destination end of the abnormal service flow, the terminal network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0219] Based on the start network element device and the end network element device, path connection processing is performed in the X-layer sub-network to obtain multiple path groups between the X-layer sub-networks.

[0220] In one implementation, any layer of subnetwork in the X-layer subnetwork is represented as the i-th layer subnetwork, where i is an integer and 1≤i≤X; the processing unit 1502 is used to perform path connection processing in the X-layer subnetwork based on the starting network element device and the end network element device, and when multiple path groups between the X-layer subnetworks are obtained, it is specifically used to:

[0221] When i=1, according to the starting network element device and the i+1th layer sub-network, multiple first path groups corresponding to the starting network element device are queried according to the border gateway protocol; the opposite end device of each first path group is a network element device in the i+1th layer sub-network;

[0222] Let i=i+1, if i=X, then according to the opposite end device and the terminal network element device of each first path group, query the Xth path group corresponding to the opposite end device of each first path group according to the border gateway protocol; the opposite end device of the Xth path group is the terminal network element device;

[0223] The multiple path groups between the X-layer sub-networks include: multiple first path groups, and X-th path groups corresponding to the opposite-end devices of each first path group.

[0224] In one implementation, the processing unit 1502 is further configured to:

[0225] Let i=i+1, if i<X, then according to the peer device of each i-1th path group in the multiple i-1th path groups and the i+1th layer subnetwork, query the multiple i-th path groups corresponding to the peer device of each i-1th path group according to the border gateway protocol;

[0226] When i=X, according to the peer device of each X-1th path group and the X-1th layer subnetwork, query the multiple X-th path groups corresponding to the peer device of each X-1th path group according to the border gateway protocol;

[0227] Let i=i-1, if i>2, then according to the peer device of each 2X-i-1 path group in the multiple 2X-i-1 path groups and the i-1 layer sub-network, query the multiple 2X-i path groups corresponding to the peer device of each i+1 path group according to the border gateway protocol; the peer device of the 2X-i path group is a network element device in the i-1 layer sub-network;

[0228] When i=2, according to the peer device and terminal network element device of each 2X-i-1 path group, query the 2X-2 path group corresponding to the peer device of each 2X-i-1 path group according to the border gateway protocol; the peer device of the 2X-2 path group is the terminal network element device;

[0229] Among them, the multiple path groups between the X-layer subnetworks include: multiple first path groups, multiple i-th path groups, multiple 2X-i path groups, and 2X-2 path groups corresponding to the opposite end devices of the 2X-i path groups.

[0230] In one implementation, the processing unit 1502 is configured to construct a directed graph according to the flow directions of the multiple path groups and the abnormal service flows between the X-layer sub-networks, specifically to:

[0231] Using multiple network element devices in multiple path groups as nodes of a directed graph;

[0232] According to the flow direction of the abnormal business flow between the X-layer sub-networks, multiple nodes are connected to obtain a directed graph.

[0233] In one implementation, the directed graph includes a start node, an end node, and X-1 intermediate layers, each intermediate layer includes one or more intermediate nodes, and any intermediate layer in the X-1 intermediate layers is represented as the jth intermediate layer, where j is an integer and 1≤j≤X-1; the processing unit 1502 is used to perform path search processing on the directed graph, and when obtaining the communication path of the abnormal service flow, it is specifically used to:

[0234] When j=1, starting from the starting node, traverse the paths between the starting node and each intermediate node in the first intermediate layer to obtain multiple first weight values ​​corresponding to the starting node; one first weight value is the weight value of the path between the starting node and an intermediate node in the first intermediate layer;

[0235] Let j=j+1, when j>X-1, calculate the Xth weight value of the path between each intermediate node and the terminal node in the j-1th intermediate layer respectively; an Xth weight value is the weight value of the path between an intermediate node and the terminal node in the j-1th intermediate layer;

[0236] Based on the multiple first weight values ​​and the Xth weight value, a shortest path is determined from the directed graph as a communication path for the abnormal business flow.

[0237] In one implementation, the processing unit 1502 is further configured to:

[0238] Let j=j+1, if j≤X-1, then starting from each intermediate node in the j-1th intermediate layer, traverse the paths between each intermediate node in the j-1th intermediate layer and each intermediate node in the j-1th intermediate layer, and obtain multiple jth weight values ​​corresponding to each intermediate node in the j-1th intermediate layer;

[0239] Based on the multiple first weight values, the multiple j-th weight values ​​corresponding to each intermediate node in the j-th intermediate layer and the X-th weight value, the shortest path is determined from the directed graph as the communication path of the abnormal business flow.

[0240] In one implementation, the acquiring unit 1501 is configured to acquire the second communication element group of the abnormal service device in the service cluster, specifically configured to:

[0241] Obtain one or more second communication element groups reported by abnormal service devices in the service cluster;

[0242] Perform data deduplication processing on one or more second communication element groups to obtain deduplicated second communication element groups.

[0243] In one implementation, the processing unit 1502 is configured to perform device analysis processing on each target network element device on the communication path, and when obtaining abnormal information of the abnormal service flow, specifically to:

[0244] Obtain one or more detection indicators; the detection indicators include at least one of the following: a traffic congestion indicator and a congestion degree indicator;

[0245] Using each detection indicator of the one or more detection indicators, performing indicator analysis and processing on each target network element device among the multiple target network element devices, and obtaining indicator information of each target network element device under each detection indicator;

[0246] According to the indicator information of each target network element device under each detection indicator, the abnormal business flow is analyzed abnormally to obtain the abnormal information of the abnormal business flow; the abnormal information of the abnormal business flow is used to describe the abnormal cause of the abnormal business flow.

[0247] In one implementation, the network element device includes a chip; the processing unit 1502 is further configured to:

[0248] Obtain a reference path, the reference path including: a service flow between a GPU card corresponding to a source end of an abnormal service flow and a GPU card corresponding to a destination end of the abnormal service flow, and multiple reference network element devices passed by the service flow; the reference path is planned by the chip based on the first communication element group of the abnormal service flow;

[0249] Compare multiple reference network element devices on the reference path with multiple target network element devices on the communication path to obtain a comparison result;

[0250] If the comparison result is that the multiple reference network element devices are the same as the multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, then triggering the step of performing device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow;

[0251] If the comparison result is: multiple reference network element devices are different from multiple target network element devices, or the communication connection sequence between multiple reference network element devices is different from the communication connection sequence between multiple target network element devices, the chip abnormality cause analysis is performed to obtain the chip abnormality information.

[0252] In one implementation, the processing unit 1502 is further configured to:

[0253] According to the abnormal information of the abnormal business flow, the business cluster is adjusted to obtain an adjusted business cluster;

[0254] Among them, there is no abnormal business flow in the adjusted business cluster.

[0255] In one implementation, the processing unit 1502 is further configured to perform cluster adjustment processing on the service cluster according to the abnormal information of the abnormal service flow, and to obtain the adjusted service cluster, specifically to:

[0256] If the abnormal cause described by the abnormal information of the abnormal business flow is that the GPU card corresponding to the source end of the abnormal business flow is faulty, the GPU card corresponding to the source end of the abnormal business flow is replaced in the business cluster to obtain a replaced business cluster.

[0257] According to one embodiment of the present application, Fig.15 The various units in the processing device of the business cluster shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the processing device of the business cluster may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it is possible to execute the following by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a memory access storage medium (RAM), and a read-only storage medium (ROM). Figure 4 and Fig.12 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.15 The processing device of the service cluster shown in the embodiment of the present application is used to implement the processing method of the service cluster. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0258] The embodiment of the present application effectively restores the communication path of the abnormal business flow when an abnormality exists in the business cluster. Compared with analyzing all business devices in the business cluster when a fault occurs, the fault investigation scope that needs to be investigated is narrowed from all business devices participating in the tasks in the business cluster to abnormal business devices that only have a communication relationship and upload the second communication element group during the abnormal time period, specifically the communication path of the abnormal business flow generated by the abnormal business device; greatly improves the processing efficiency when a fault occurs in the business cluster, avoids the waste of computing power of the business devices in the business cluster, and saves the cost of task execution.

[0259] Fig.16 FIG. 1 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Fig.16 , the computer device includes a processor 1601, a communication interface 1602 and a computer-readable storage medium 1603. The processor 1601, the communication interface 1602 and the computer-readable storage medium 1603 can be connected via a bus or other means. The communication interface 1602 is used to receive and send data. The computer-readable storage medium 1603 can be stored in the memory of the computer device, the computer-readable storage medium 1603 is used to store a computer program, the computer program includes program instructions, and the processor 1601 is used to execute the program instructions stored in the computer-readable storage medium 1603. The processor 1601 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0260] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor 1601 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0261] In one embodiment, one or more instructions are stored in the computer-readable storage medium; the processor 1601 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned service cluster processing method embodiment; the service cluster includes multiple service devices and multiple network element devices, and any two service devices are communicated and connected through at least one network element device; each service device includes at least one GPU card; in a specific implementation, the one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and the following steps are executed:

[0262] Obtain a first communication element group of multiple service flows in a service cluster, where a service flow refers to a data flow from a GPU card in a service device at a source end to a GPU card in a service device at a destination end; the first communication element group includes an identifier of a GPU card corresponding to a source end of the service flow and an identifier of a GPU card corresponding to a destination end;

[0263] Acquire a second communication element group of an abnormal service device in the service cluster, where the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card;

[0264] Based on the first communication element group and the second communication element group, screening out abnormal service flows from the service cluster;

[0265] Determine the communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, wherein the communication path includes multiple target network element devices that the abnormal service flow passes through in sequence;

[0266] Perform device analysis and processing on each target network element device on the communication path to obtain abnormal information of abnormal service flows.

[0267] In one implementation, the service cluster includes an X-level network, the X-level network includes X-layer subnetworks distributed in a hierarchical manner, each layer of the subnetwork includes multiple network element devices, X is an integer and X≥2; one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when executing, according to the GPU card corresponding to the source end of the abnormal service flow and the GPU card corresponding to the destination end, determine the communication path of the abnormal service flow, and specifically perform the following steps:

[0268] According to the GPU card corresponding to the source end of the abnormal business flow, the GPU card corresponding to the destination end of the abnormal business flow, and the network element devices in each layer of the sub-network in the X-level networking, multiple path groups between the X-layer sub-networks are obtained; each path group includes one or more equal-cost paths, and the two path endpoints of the equal-cost path are respectively network element devices in two adjacent layers of sub-networks;

[0269] According to the flow directions of multiple path groups and abnormal business flows between the X-layer sub-networks, a directed graph is constructed;

[0270] Perform path search processing on the directed graph to obtain the communication path of the abnormal business flow.

[0271] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when executing, according to the GPU card corresponding to the source end of the abnormal service flow, the GPU card corresponding to the destination end of the abnormal service flow, and the network element device in each layer of the sub-network in the X-level networking, multiple path groups between the X-layer sub-networks are obtained, and the following steps are specifically performed:

[0272] According to the identification of the GPU card corresponding to the source end of the abnormal service flow, the starting network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0273] According to the identification of the GPU card corresponding to the destination end of the abnormal service flow, the terminal network element device of the abnormal service flow is determined in the first layer subnetwork in the X-level networking;

[0274] Based on the start network element device and the end network element device, path connection processing is performed in the X-layer sub-network to obtain multiple path groups between the X-layer sub-networks.

[0275] In one implementation, any layer of subnetwork in the X-layer subnetwork is represented as the i-th layer subnetwork, where i is an integer and 1≤i≤X; one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and executed based on the starting network element device and the end network element device, and path connection processing is performed in the X-layer subnetwork to obtain multiple path groups between the X-layer subnetworks, and the following steps are specifically performed:

[0276] When i=1, according to the starting network element device and the i+1th layer sub-network, multiple first path groups corresponding to the starting network element device are queried according to the border gateway protocol; the opposite end device of each first path group is a network element device in the i+1th layer sub-network;

[0277] Let i=i+1, if i=X, then according to the opposite end device and the terminal network element device of each first path group, query the Xth path group corresponding to the opposite end device of each first path group according to the border gateway protocol; the opposite end device of the Xth path group is the terminal network element device;

[0278] The multiple path groups between the X-layer sub-networks include: multiple first path groups, and X-th path groups corresponding to the opposite-end devices of each first path group.

[0279] In one implementation, one or more instructions in the computer readable storage medium are loaded by the processor 1601 and further perform the following steps:

[0280] Let i=i+1, if i<X, then according to the peer device of each i-1th path group in the multiple i-1th path groups and the i+1th layer subnetwork, query the multiple i-th path groups corresponding to the peer device of each i-1th path group according to the border gateway protocol;

[0281] When i=X, according to the peer device of each X-1th path group and the X-1th layer subnetwork, query the multiple X-th path groups corresponding to the peer device of each X-1th path group according to the border gateway protocol;

[0282] Let i=i-1, if i>2, then according to the peer device of each 2X-i-1 path group in the multiple 2X-i-1 path groups and the i-1 layer sub-network, query the multiple 2X-i path groups corresponding to the peer device of each i+1 path group according to the border gateway protocol; the peer device of the 2X-i path group is a network element device in the i-1 layer sub-network;

[0283] When i=2, according to the peer device and terminal network element device of each 2X-i-1 path group, query the 2X-2 path group corresponding to the peer device of each 2X-i-1 path group according to the border gateway protocol; the peer device of the 2X-2 path group is the terminal network element device;

[0284] Among them, the multiple path groups between the X-layer subnetworks include: multiple first path groups, multiple i-th path groups, multiple 2X-i path groups, and 2X-2 path groups corresponding to the opposite end devices of the 2X-i path groups.

[0285] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when executing to construct a directed graph according to multiple path groups and the flow direction of abnormal service flows between the X-layer sub-networks, the following steps are specifically performed:

[0286] Using multiple network element devices in multiple path groups as nodes of a directed graph;

[0287] According to the flow direction of the abnormal business flow between the X-layer sub-networks, multiple nodes are connected to obtain a directed graph.

[0288] In one implementation, the directed graph includes a start node, an end node, and X-1 intermediate layers, each intermediate layer includes one or more intermediate nodes, and any intermediate layer in the X-1 intermediate layers is represented as the jth intermediate layer, where j is an integer and 1≤j≤X-1; one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when performing a path search process on the directed graph to obtain a communication path of an abnormal service flow, the following steps are specifically performed:

[0289] When j=1, starting from the starting node, traverse the paths between the starting node and each intermediate node in the first intermediate layer to obtain multiple first weight values ​​corresponding to the starting node; one first weight value is the weight value of the path between the starting node and an intermediate node in the first intermediate layer;

[0290] Let j=j+1, when j>X-1, calculate the Xth weight value of the path between each intermediate node and the terminal node in the j-1th intermediate layer respectively; an Xth weight value is the weight value of the path between an intermediate node and the terminal node in the j-1th intermediate layer;

[0291] Based on the multiple first weight values ​​and the Xth weight value, a shortest path is determined from the directed graph as a communication path for the abnormal business flow.

[0292] In one implementation, one or more instructions in the computer readable storage medium are loaded by the processor 1601 and further perform the following steps:

[0293] Let j=j+1, if j≤X-1, then starting from each intermediate node in the j-1th intermediate layer, traverse the paths between each intermediate node in the j-1th intermediate layer and each intermediate node in the j-1th intermediate layer, and obtain multiple jth weight values ​​corresponding to each intermediate node in the j-1th intermediate layer;

[0294] Based on the multiple first weight values, the multiple j-th weight values ​​corresponding to each intermediate node in the j-th intermediate layer and the X-th weight value, the shortest path is determined from the directed graph as the communication path of the abnormal business flow.

[0295] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when executing to obtain the second communication element group of the abnormal service device in the service cluster, the following steps are specifically performed:

[0296] Obtain one or more second communication element groups reported by abnormal service devices in the service cluster;

[0297] Perform data deduplication processing on one or more second communication element groups to obtain deduplicated second communication element groups.

[0298] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when performing device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow, the following steps are specifically performed:

[0299] Obtain one or more detection indicators; the detection indicators include at least one of the following: a traffic congestion indicator and a congestion degree indicator;

[0300] Using each detection indicator of the one or more detection indicators, performing indicator analysis and processing on each target network element device among the multiple target network element devices, and obtaining indicator information of each target network element device under each detection indicator;

[0301] According to the indicator information of each target network element device under each detection indicator, the abnormal business flow is analyzed abnormally to obtain the abnormal information of the abnormal business flow; the abnormal information of the abnormal business flow is used to describe the abnormal cause of the abnormal business flow.

[0302] In one implementation, the network element device includes a chip; one or more instructions in a computer-readable storage medium are loaded by the processor 1601 and further perform the following steps:

[0303] Obtain a reference path, the reference path including: a service flow between a GPU card corresponding to a source end of an abnormal service flow and a GPU card corresponding to a destination end of the abnormal service flow, and multiple reference network element devices passed by the service flow; the reference path is planned by the chip based on the first communication element group of the abnormal service flow;

[0304] Compare multiple reference network element devices on the reference path with multiple target network element devices on the communication path to obtain a comparison result;

[0305] If the comparison result is that the multiple reference network element devices are the same as the multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, then triggering the step of performing device analysis and processing on each target network element device on the communication path to obtain abnormal information of the abnormal service flow;

[0306] If the comparison result is: multiple reference network element devices are different from multiple target network element devices, or the communication connection sequence between multiple reference network element devices is different from the communication connection sequence between multiple target network element devices, the chip abnormality cause analysis is performed to obtain the chip abnormality information.

[0307] In one implementation, one or more instructions in the computer readable storage medium are loaded by the processor 1601 and further perform the following steps:

[0308] According to the abnormal information of the abnormal business flow, the business cluster is adjusted to obtain an adjusted business cluster;

[0309] Among them, there is no abnormal business flow in the adjusted business cluster.

[0310] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1601 and when performing cluster adjustment processing on the service cluster according to the abnormal information of the abnormal service flow to obtain the adjusted service cluster, the following steps are specifically performed:

[0311] If the abnormal cause described by the abnormal information of the abnormal business flow is that the GPU card corresponding to the source end of the abnormal business flow is faulty, the GPU card corresponding to the source end of the abnormal business flow is replaced in the business cluster to obtain a replaced business cluster.

[0312] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the processing method of the business cluster in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0313] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned service cluster processing method.

[0314] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0315] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0316] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field within the technical scope disclosed in the present application can be easily thought of for changes or replacements, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for processing a service cluster, characterized in that: The service cluster includes a plurality of service devices and a plurality of network element devices, and any two of the service devices are communicatively connected via at least one of the network element devices; each of the service devices includes at least one GPU card; and the method includes: Obtain a first communication element group of multiple business flows in the business cluster, where the business flow refers to a data flow from a GPU card in the business device at the source end to a GPU card in the business device at the destination end; the first communication element group includes an identifier of the GPU card corresponding to the source end of the business flow and an identifier of the GPU card corresponding to the destination end; Acquire a second communication element group of an abnormal service device in the service cluster, where the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card; Based on the first communication element group and the second communication element group, screening out abnormal service flows from the service cluster; Determine a communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, wherein the communication path includes a plurality of target network element devices that the abnormal service flow passes through in sequence; Device analysis and processing are performed on each of the target network element devices on the communication path to obtain abnormal information of the abnormal service flow.

2. The method according to claim 1, characterized in that The service cluster includes an X-level network, the X-level network includes a hierarchically distributed X-layer subnetwork, each layer of the subnetwork includes a plurality of network element devices, X is an integer and X≥2; the determining the communication path of the abnormal service flow according to the GPU card corresponding to the source end of the abnormal service flow and the GPU card corresponding to the destination end includes: According to the GPU card corresponding to the source end of the abnormal business flow, the GPU card corresponding to the destination end of the abnormal business flow, and the network element devices in each layer of sub-network in the X-level networking, multiple path groups between the X-layer sub-networks are obtained; each path group includes one or more equal-cost paths, and two path endpoints of the equal-cost path are respectively network element devices in two adjacent layers of sub-networks; Constructing a directed graph according to the plurality of path groups and the flow direction of the abnormal service flow between the X-layer subnetworks; Perform path search processing on the directed graph to obtain the communication path of the abnormal business flow.

3. The method according to claim 2, characterized in that The method of obtaining multiple path groups between the X-layer subnetworks according to the GPU card corresponding to the source end of the abnormal service flow, the GPU card corresponding to the destination end of the abnormal service flow, and the network element devices in each layer of the subnetwork in the X-level networking includes: According to the identifier of the GPU card corresponding to the source end of the abnormal service flow, determining the starting network element device of the abnormal service flow in the first layer subnetwork in the X-level networking; According to the identifier of the GPU card corresponding to the destination end of the abnormal service flow, determining the terminal network element device of the abnormal service flow in the first layer subnetwork in the X-level networking; Based on the starting network element device and the end network element device, path connection processing is performed in the X-layer subnetwork to obtain multiple path groups between the X-layer subnetworks.

4. The method according to claim 3, characterized in that Any layer of subnetwork in the X-layer subnetwork is represented as the i-th layer subnetwork, where i is an integer and 1≤i≤X; performing path connection processing in the X-layer subnetwork based on the starting network element device and the end network element device to obtain multiple path groups between the X-layer subnetworks includes: When i=1, according to the starting network element device and the i+1th layer subnetwork, multiple first path groups corresponding to the starting network element device are queried according to the border gateway protocol; the opposite end device of each first path group is a network element device in the i+1th layer subnetwork; Let i=i+1, if i=X, then according to the opposite end device of each first path group and the terminal network element device, query the Xth path group corresponding to the opposite end device of each first path group according to the border gateway protocol; the opposite end device of the Xth path group is the terminal network element device; The multiple path groups between the X-layer sub-networks include: the multiple first path groups, and the X-th path groups corresponding to the opposite-end devices of each first path group.

5. The method according to claim 4, characterized in that The method further comprises: Let i=i+1, if i<X, then according to the peer device of each i-1th path group in the multiple i-1th path groups and the i+1th layer subnetwork, query the multiple i-th path groups corresponding to the peer device of each i-1th path group according to the border gateway protocol; When i=X, according to the peer device of each X-1th path group and the X-1th layer subnetwork, query the multiple X-th path groups corresponding to the peer device of each X-1th path group according to the border gateway protocol; Let i=i-1, if i>2, then according to the peer device of each 2X-i-1 path group in the multiple 2X-i-1 path groups and the i-1 layer sub-network, query the multiple 2X-i path groups corresponding to the peer device of each i+1 path group according to the border gateway protocol; the peer device of the 2X-i path group is a network element device in the i-1 layer sub-network; When i=2, according to the opposite end device of each 2X-i-1 path group and the terminal network element device, query the 2X-2 path group corresponding to the opposite end device of each 2X-i-1 path group according to the border gateway protocol; the opposite end device of the 2X-2 path group is the terminal network element device; Among them, the multiple path groups between the X-layer subnetworks include: the multiple first path groups, the multiple i-th path groups, the multiple 2X-i path groups and the 2X-2 path groups corresponding to the opposite end devices of the 2X-i path groups respectively.

6. The method according to any one of claims 2 to 5, characterized in that: The step of constructing a directed graph according to the plurality of path groups and the flow direction of the abnormal service flow between the X-layer subnetworks includes: Using multiple network element devices in the multiple path groups as nodes of a directed graph; According to the flow direction of the abnormal service flow between the X-layer sub-networks, multiple nodes are connected to obtain a directed graph.

7. The method according to claim 6, characterized in that The directed graph includes a starting node, an end node and X-1 intermediate layers, each intermediate layer includes one or more intermediate nodes, and any intermediate layer of the X-1 intermediate layers is represented as the jth intermediate layer, where j is an integer and 1≤j≤X-1; performing path search processing on the directed graph to obtain the communication path of the abnormal service flow includes: When j=1, starting from the starting node, traverse the paths between the starting node and each intermediate node in the first intermediate layer to obtain multiple first weight values ​​corresponding to the starting node; one first weight value is the weight value of the path between the starting node and an intermediate node in the first intermediate layer; Let j=j+1, when j>X-1, calculate the Xth weight value of the path between each intermediate node in the j-1th intermediate layer and the terminal node respectively; an Xth weight value is the weight value of the path between an intermediate node in the j-1th intermediate layer and the terminal node; Based on the multiple first weight values ​​and the Xth weight value, a shortest path is determined from the directed graph as a communication path for the abnormal business flow.

8. The method according to claim 7, characterized in that The method further comprises: Let j=j+1, if j≤X-1, then starting from each intermediate node in the j-1th intermediate layer, traverse the paths between each intermediate node in the j-1th intermediate layer and each intermediate node in the j-1th intermediate layer, and obtain multiple j-th weight values ​​corresponding to each intermediate node in the j-1th intermediate layer; Based on the multiple first weight values, the multiple j-th weight values ​​corresponding to each intermediate node in the j-th intermediate layer and the X-th weight value, the shortest path is determined from the directed graph as the communication path of the abnormal business flow.

9. The method according to claim 1, characterized in that The obtaining of a second communication element group of abnormal service devices in the service cluster includes: Obtaining one or more second communication element groups reported by abnormal service devices on the service cluster; Perform data deduplication processing on the one or more second communication element groups to obtain the second communication element groups after deduplication processing.

10. The method according to claim 1, characterized in that The performing device analysis and processing on each of the target network element devices on the communication path to obtain abnormal information of the abnormal service flow includes: Obtain one or more detection indicators; the detection indicators include at least one of the following: a traffic congestion indicator and a congestion degree indicator; Using each of the one or more detection indicators, perform indicator analysis and processing on each of the multiple target network element devices to obtain indicator information of each target network element device under each of the detection indicators; According to the indicator information of each target network element device under each of the detection indicators, the abnormal business flow is analyzed abnormally to obtain the abnormal information of the abnormal business flow; the abnormal information of the abnormal business flow is used to describe the abnormal cause of the abnormal business flow.

11. The method according to claim 1 or 10, characterized in that: The network element device includes a chip; before performing device analysis and processing on each of the target network element devices on the communication path to obtain the abnormal information of the abnormal service flow, it also includes: Acquire a reference path, wherein the reference path includes: a service flow between the GPU card corresponding to the source end of the abnormal service flow and the GPU card corresponding to the destination end of the abnormal service flow, and a plurality of reference network element devices passed by the service flow; the reference path is planned by the chip based on the first communication element group of the abnormal service flow; Compare the multiple reference network element devices on the reference path with the multiple target network element devices on the communication path to obtain a comparison result; If the comparison result is that the multiple reference network element devices are the same as the multiple target network element devices, and the communication connection sequence between the multiple reference network element devices is the same as the communication connection sequence between the multiple target network element devices, then triggering the step of performing device analysis processing on each of the target network element devices on the communication path to obtain abnormal information of the abnormal service flow; If the comparison result is: the multiple reference network element devices are not the same as the multiple target network element devices, or the communication connection sequence between the multiple reference network element devices is not the same as the communication connection sequence between the multiple target network element devices, then the cause of the abnormality of the chip is analyzed to obtain the abnormality information of the chip.

12. The method according to claim 1, characterized in that The method further comprises: According to the abnormal information of the abnormal service flow, the service cluster is adjusted to obtain the adjusted service cluster; Among them, the abnormal service flow does not exist in the adjusted service cluster.

13. The method according to claim 12, characterized in that The performing cluster adjustment processing on the service cluster according to the abnormal information of the abnormal service flow to obtain the adjusted service cluster includes: If the abnormal cause described in the abnormal information of the abnormal business flow is that the GPU card corresponding to the source end of the abnormal business flow is faulty, the GPU card corresponding to the source end of the abnormal business flow is replaced in the business cluster to obtain the replaced business cluster.

14. A processing device for a service cluster, characterized in that: The service cluster includes a plurality of service devices and a plurality of network element devices, and any two of the service devices are connected to each other by communication via at least one of the network element devices; each of the service devices includes at least one GPU card; the apparatus includes: An acquisition unit is used to acquire a first communication element group of multiple business flows in the business cluster, where the business flow refers to a data flow from a GPU card in the business device at the source end to a GPU card in the business device at the destination end; the first communication element group includes an identifier of the GPU card corresponding to the source end of the business flow and an identifier of the GPU card corresponding to the destination end; The acquisition unit is further used to acquire a second communication element group of the abnormal service device in the service cluster, wherein the second communication element group includes an identifier of an abnormal GPU card in the abnormal service device and an identifier of a GPU card communicating with the abnormal GPU card; A processing unit, configured to filter out abnormal service flows from the service cluster based on the first communication element group and the second communication element group; The processing unit is further used to determine a communication path of the abnormal service flow according to the GPU card corresponding to the source end and the GPU card corresponding to the destination end of the abnormal service flow, wherein the communication path includes multiple target network element devices that the abnormal service flow passes through in sequence; The processing unit is further used to perform device analysis processing on each of the target network element devices on the communication path to obtain abnormal information of the abnormal service flow.

15. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method for processing a service cluster according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the service cluster processing method according to any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product includes computer instructions, and when the computer instructions are executed by a processor, the method for processing a service cluster according to any one of claims 1 to 13 is implemented.

Images