Computing node detection method, system and device in AI computing power cluster network

Abstract

The invention discloses a method, a system and a device for detecting computing nodes in an AI (Artificial Intelligence) computing power cluster network, which are used for carrying out anomaly detection on the computing nodes in the AI computing power cluster network before a neural network model adopted for executing an AI-related computing task is trained. The anomaly detection process comprises the following steps of: adopting multiple rounds of anomaly detection, sequentially screening a detection group with an abnormal computing node from the AI computing power cluster network, screening a sub-detection group with the abnormal computing node from the detection group with the abnormal computing node until the number of the computing nodes in the sub-detection group with the abnormal computing node is 1, and detecting an abnormal computing node. And in each round of detection, dividing a neural network model for detection into a plurality of operators, configuring the operators on computing nodes in the detection group or the sub-detection group for training, and judging whether the detection group or the sub-detection group is abnormal or not based on a training result obtained by training. In this way, the computing nodes in the AI computing power cluster network are detected.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a method, system and device for detecting computing nodes in an artificial intelligence (AI) computing power cluster network. Background Art

[0002] The AI ​​computing power cluster network consists of multiple computing nodes, which are interconnected to form a computing resource pool to synchronously execute AI-related computing tasks. Before executing a computing task, the AI ​​computing power cluster network needs to train the neural network model used to execute the computing task. During the training process, different operators of the neural network model are configured on different computing nodes, and each configured computing node trains the corresponding operator to ensure the consistency of the parameters in the neural network model when the computing task is distributed in the AI ​​computing power cluster network.

[0003] When the AI ​​computing power cluster network trains the neural network model, the hardware failure of the computing nodes and / or the computing rate of the software play a decisive role in the training effect. Therefore, before the AI ​​computing power cluster network trains the neural network model, it is necessary to perform anomaly detection on the computing nodes in the AI ​​computing power cluster network, detect the computing nodes with hardware failures and / or asynchronous software computing rates, and perform fault tolerance processing. Summary of the invention

[0004] An embodiment of the present application provides a method for detecting computing nodes in an AI computing power cluster network, which can detect computing nodes in an AI computing power cluster network.

[0005] The embodiments of the present application also provide a node detection system and device in an AI computing power cluster network, which can detect computing nodes in the AI ​​computing power cluster network.

[0006] The present application embodiment provides a computing node detection method in an AI computing power cluster network, including:

[0007] Divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, and the first detection group is used as the current detection group;

[0008] configuring at least one operator in the neural network model to the computing nodes of the current detection group respectively, so that the computing nodes of the current detection group train the corresponding operators to obtain the training results of the current detection group;

[0009] Based on the training result of the current detection group, determining that the current detection group is abnormal;

[0010] When it is determined that the number of computing nodes in the current detection group is not 1, the computing nodes in the current detection group are divided into at least one second detection group, wherein the second detection group includes 2 n computing nodes, where n is a natural number, taking the second detection group as the current detection group, returning to the step of configuring at least one operator in the neural network model to the computing nodes of the current detection group and subsequent steps, and continuing to execute;

[0011] When it is determined that the number of computing nodes in the current detection group is 1, it is determined that the computing nodes in the current detection group are abnormal.

[0012] In the above method, dividing the computing nodes in the AI ​​computing power cluster network into more than one first detection group includes:

[0013] Evenly divide the computing nodes in the AI ​​computing power cluster network into the first detection group;

[0014] The step d of dividing the computing nodes in the current detection group into at least one second detection group includes:

[0015] The computing nodes in the current detection group are evenly divided into the second detection group.

[0016] In the above method, configuring at least one operator in the neural network model to the computing nodes of the current detection group includes:

[0017] Based on the PP mode of the preset pipeline parallel PP number and the TP mode of the preset tensor parallel TP number, the neural network model is divided into operators, and the divided operators are respectively configured to the computing nodes in the current detection group;

[0018] In step b, enabling the computing nodes of the current detection group to train corresponding operators includes:

[0019] Based on the DP method with a preset number of data parallel DPs, the training data used by the neural network model is divided and input in parallel to the corresponding operators of the computing nodes of the current detection group for training.

[0020] In the above method, the step of enabling the computing nodes of the current detection group to train corresponding operators further includes:

[0021] According to a preset number of training iterations, the corresponding operators are iteratively trained on the computing nodes of the current detection group.

[0022] In the above method, the computing node of the current detection group is a graphics processing unit (GPU) server, and the GPU server includes at least one GPU accelerator card;

[0023] The obtaining of the training result of the current detection group includes:

[0024] The loss function value and / or the performance value, wherein the performance value is the number of tokens executed per second divided by the number of GPU accelerator cards tgs in the computing nodes of the current detection group;

[0025] Determining that the current detection group is abnormal in step c includes:

[0026] Comparing the loss function value with a preset loss function standard value to determine that the current detection group is an accuracy abnormality detection group; or / and,

[0027] The performance value is compared with a preset performance standard value golden to determine that the current detection group is a performance abnormality detection group.

[0028] In the above method, determining that the current detection group is an accuracy abnormality detection group includes:

[0029] For the current detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed a preset first threshold;

[0030] Based on the current detection group, the number of loss function values ​​exceeding the first threshold is summarized and determined, and it is judged that it is greater than a preset second threshold, and the current detection group is determined to be an accuracy abnormality detection group.

[0031] In the above method, determining that the current detection group is a performance abnormality detection group includes:

[0032] Using the performance value of the current detection group and taking golden as the benchmark, calculate the absolute value of the variance of the current detection group;

[0033] It is determined whether the absolute value of the variance of the current detection group exceeds a preset ratio value of a set variance reference value. If so, it is determined that the current detection group is a performance abnormality detection group.

[0034] The embodiment of the present application also provides a computing node detection device in an AI computing power cluster network, including: a division unit, a configuration unit, an interaction unit and a detection unit, wherein:

[0035] The division unit is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; under the instruction of the detection unit, the computing nodes in the current detection group are divided into at least one second detection group, the second detection group includes 2n computing nodes, n is a natural number, and the second detection group is used as the current detection group;

[0036] The configuration unit is used to configure at least one operator in the neural network model to the computing nodes of the current detection group respectively;

[0037] The interaction unit is used to interact with the computing nodes of the current detection group, so that the computing nodes of the current detection group train corresponding operators and obtain training results of the current detection group;

[0038] The detection unit is used to determine whether the current detection group is abnormal based on the training result of the current detection group; determine whether the number of computing nodes in the current detection group is 1, and if not, instruct the division unit to divide the current detection group; if yes, determine that the computing nodes in the current detection group are abnormal.

[0039] This embodiment also provides a computing node detection system in an AI computing power cluster network, including: a detection node and multiple computing nodes, wherein:

[0040] The detection node is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; at least one operator in the neural network model is respectively configured to the computing nodes of the current detection group; interacting with the computing nodes of the current detection group so that the computing nodes of the current detection group train the corresponding operators and obtain the training results of the current detection group; based on the training results of the current detection group, determining that the current detection group is abnormal; determining whether the number of computing nodes in the current detection group is 1, if not, dividing the computing nodes in the current detection group into at least one second detection group, the second detection group includes 2 n computing nodes, n is a natural number, taking the second detection group as the current detection group, and continuing to execute the above configuration process; if yes, determining that the computing nodes in the current detection group are abnormal;

[0041] The computing node is used to configure the corresponding operator in the neural network model under the configuration of the detection node; under the instruction of the detection node, train the corresponding operator in the neural network model and send the obtained training results to the detection node.

[0042] An embodiment of the present application also provides an electronic device, including a processor, wherein the processor is used to execute a computing node detection method in an AI computing power cluster network as described in any one of the above.

[0043] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the processor executes a computing node detection method in an AI computing power cluster network as described in any one of the above items.

[0044] As can be seen above, the embodiment of the present application performs abnormality detection on the computing nodes in the AI ​​computing power cluster network before training the neural network model used to perform AI-related computing tasks in the AI ​​computing power cluster network. The abnormality detection process includes: using multiple rounds of abnormality detection, successively screening the detection group with abnormal computing nodes from the AI ​​computing power cluster network, screening the sub-detection group with abnormal computing nodes from the detection group with abnormal computing nodes, until the number of computing nodes in the sub-detection group with abnormal computing nodes is 1, and detecting the abnormal computing nodes in the AI ​​computing power cluster network. In each round of abnormality detection, the neural network model used for detection is divided into multiple operators, configured on the computing nodes in the detection group or sub-detection group, and trained, and judging whether the detection group or sub-detection group is abnormal based on the training results obtained by the training. In this way, the abnormal computing nodes can be determined in the AI ​​computing power cluster network, and the abnormal computing nodes can be fault-tolerant later, or based on the abnormal computing nodes, before the neural network model used to perform AI-related computing tasks is trained, the training strategy of the neural network model is updated to avoid wasting the computing power resources of the AI ​​computing power cluster network. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a computing node detection method in an AI computing power cluster network provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of a computing node detection device in an AI computing power cluster network provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the structure of a computing node detection system in an AI computing power cluster network provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the overall process of detecting a GPU server provided for a specific example of this application;

[0049] Figure 5 A flow chart of a method for detecting abnormal GPU servers provided for a specific example of this application;

[0050] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0054] When the AI ​​computing cluster network trains the neural network model, each computing node needs to synchronously train the corresponding operator to ensure the consistency of the parameters in the neural network model when the training task is distributed in the AI ​​computing cluster network, and the consistency of the subsequent distributed computing tasks. If there are computing synchronization problems between computing nodes, it may cause inconsistent iterations of the neural network model during the training process, thereby affecting the training effect. The reasons for the synchronization problems between computing nodes are: hardware failure of the computing node, software execution error, or slow computing rate of the software, which causes the computing node to become abnormal.

[0055] Hardware failures of computing nodes, such as damage to the video memory or internal computing processors, may cause training failures or produce erroneous training results, directly affecting the convergence and accuracy of the neural network model. In some cases, when the computing nodes in the AI ​​computing cluster network use graphics processing unit (GPU) servers, their computing accuracy is different from that of networks composed of nodes with other types of processors, resulting in inconsistent accuracy of neural network models when trained in different networks. If there are GPU computing nodes with slow computing speeds in the AI ​​computing cluster network, it will also slow down the progress of the entire training process, resulting in a "bottleneck" effect.

[0056] Therefore, effective anomaly detection and fault tolerance of computing nodes in the AI ​​computing power cluster network can reduce the impact of computing nodes on training accuracy. Effective anomaly detection of computing nodes can quickly detect abnormal computing nodes in the AI ​​computing power cluster network and adopt preset fault tolerance methods to recover, such as reallocating operators of the neural network model to other non-abnormal computing nodes, or restarting the computing node to repair the hardware failure of the computing node.

[0057] Furthermore, when abnormal computing nodes are detected in the AI ​​computing cluster network, in order to reduce the impact of abnormal computing nodes on the training of the neural network model, the training strategy of the neural network model can be adjusted during the subsequent training of the neural network model, such as configuring each operator of the neural network model and the corresponding training data in parallel to non-abnormal computing nodes for training, or using more complex load balancing configuration methods for configuration.

[0058] Therefore, in order to improve the utilization rate of the AI ​​computing power cluster network in executing AI-related computing tasks, the embodiment of the present application performs abnormality detection on the computing nodes therein before training the neural network model used to execute the computing task, and detects abnormal computing nodes. Subsequently, fault tolerance is performed on the abnormal computing nodes, or the training strategy of the neural network model is updated based on the abnormal computing nodes to avoid the AI ​​computing power cluster network discovering the abnormality of the computing nodes during the training process, thereby wasting computing power resources.

[0059] Figure 1 A flowchart of a computing node detection method in an AI computing power cluster network provided in an embodiment of the present application, wherein the specific steps include:

[0060] Step 101: divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, and the first detection group is used as the current detection group;

[0061] Step 102: configure at least one operator in the neural network model to the computing nodes of the current detection group, so that the computing nodes of the current detection group train the corresponding operators to obtain the training results of the current detection group;

[0062] Step 103: Based on the training result of the current detection group, determine whether the current detection group is abnormal;

[0063] Step 104: determine whether the number of computing nodes in the current detection group is 1, if not, execute step 105; if not, execute step 106;

[0064] Step 105: divide the computing nodes in the current detection group into at least one second detection group, the second detection group including 2 n computing nodes, where n is a natural number, the second detection group is used as the current detection group, and the process returns to step 102 to continue execution;

[0065] Step 106: Determine whether a computing node in the current detection group is abnormal.

[0066] It can be seen from the above method that the present application adopts multiple rounds of anomaly detection, successively screening detection groups with abnormal computing nodes from the AI ​​computing power cluster network, and screening sub-detection groups with abnormal computing nodes from the detection groups with abnormal computing nodes, until the number of computing nodes in the sub-detection group with abnormal computing nodes is 1, and the abnormal computing nodes in the AI ​​computing power cluster network are detected. In each round of anomaly detection, the neural network model used for detection is divided into multiple operators, configured on the computing nodes in the detection group or sub-detection group, and trained, and whether the detection group or sub-detection group is abnormal is judged based on the training results obtained by the training. Here, for different rounds of anomaly detection, the neural network model used for detection is the same or different, and the number of iterations of the neural network model is the same or different.

[0067] The neural network model in step 102 of the above method is a neural network model used for detection, which is selected from existing neural network models as needed. The configuration of at least one operator in the neural network model to the computing nodes of the current detection group in step 102 includes:

[0068] Based on the preset pipeline parallel (PP) number and the preset tensor parallel (TP) number, the neural network model is divided into operators, and the divided operators are respectively configured to the computing nodes of the current detection group.

[0069] In step 102 of the above method, training the corresponding operator of the computing node of the current detection group includes:

[0070] Based on a DP method with a preset data parallel (DP) number, the training data used by the neural network model is divided and input in parallel to the corresponding operators of the computing nodes of the current detection group for training.

[0071] It can be seen that the embodiment of the present application divides the neural network model and training data into PP, TP and DP modes when configuring the current detection group and in the current detection group, thereby improving the detection efficiency and detection accuracy.

[0072] Before determining that the current detection group is abnormal in step 103 in the above method, it is further included to determine whether the current detection group is abnormal. If so, step 103 and subsequent steps are executed; if not, the computing nodes in the current detection group are all normal, and the above method ends.

[0073] In an embodiment, there are multiple first detection groups in the AI ​​computing power cluster network, and the above method is performed for each first detection group to determine one or more abnormal first detection groups in the AI ​​computing power cluster network. There are one or more second detection groups in the AI ​​computing power cluster network, and the above method is performed for each second detection group to determine one or more abnormal second detection groups in the AI ​​computing power cluster network, and then the subsequent second detection groups are divided again and abnormal detection is performed accordingly until the abnormal computing nodes are determined.

[0074] In step 106 of the above method, an abnormal computing node is determined, and a preset fault-tolerant method can be used to process it so that it can operate normally and be consistent with the computing rate of other computing nodes in the AI ​​computing power cluster network, and participate in the training process of the neural network model used in the subsequent execution of AI-related computing tasks. Among them, the fault-tolerant method includes: reallocating the operators of the neural network model to other non-abnormal computing nodes, or restarting the computing node.

[0075] Alternatively, in step 106 of the above method, an abnormal computing node is determined, and before the training process of the neural network model used for subsequent execution of AI-related computing tasks, the preset training strategy of the neural network model is updated, and the updated training strategy is: each operator of the neural network model and the corresponding training data are configured in parallel on non-abnormal computing nodes.

[0076] By using the above two processing methods, the impact of abnormal computing nodes on the training of neural network models used in AI computing cluster network training to perform AI-related computing tasks can be reduced, thereby improving training accuracy and training speed.

[0077] In the above method, dividing the computing nodes in the neural network model into more than one first detection group includes: dividing the computing nodes in the AI ​​computing power cluster network into the first detection group on an average basis. Dividing the computing nodes in the current detection group into at least one second detection group includes: dividing the computing nodes in the current detection group into the second detection group on an average basis. In this way, the balance of abnormal detection of computing nodes in the AI ​​computing power cluster network is ensured, and the detection efficiency and detection accuracy are improved.

[0078] Specifically, evenly dividing the computing nodes in the AI ​​computing power cluster network into the first detection group includes: dividing 4 computing nodes into the first detection group; evenly dividing the computing nodes in the current detection group into the second detection group includes: dividing 2 computing nodes into the second detection group, or dividing 1 computing node into the second detection group. Here, the computing nodes are GPU servers, and each GPU server has 8 built-in GPU accelerator cards. Of course, each GPU server can also have other numbers of GPU accelerator cards built in, which is not limited here.

[0079] In step 102 of the above method, the step of enabling the computing nodes of the current detection group to train the corresponding operators includes:

[0080] According to the preset number of training iterations, the corresponding operator is trained on the computing node of the current detection group. Here, the number of training iterations is set according to the training requirements of the neural network model and is not limited here. The number of training iterations is related to the training accuracy of the neural network model and the accuracy of the anomaly detection of the computing node. Therefore, in order to make the anomaly detection of the computing node more accurate, the number of training iterations can be set to be relatively large.

[0081] In step 102 of the above method, the training results of the current detection group include:

[0082] The loss function value and / or the performance value, wherein the performance value is the amount of tokens executed per second divided by the number of GPU accelerator cards in the computing nodes of the current detection group (tgs, token / GPU / second).

[0083] In step 103 of the above method, determining that the current detection group is abnormal includes:

[0084] Comparing the loss function value with a preset loss function standard value to determine that the current detection group is an accuracy abnormality detection group; or / and,

[0085] The performance value is compared with a preset performance standard value (golden) to determine that the current detection group is a performance abnormality detection group.

[0086] Specifically, determining that the current detection group is an accuracy abnormality detection group includes:

[0087] For the current detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed a preset first threshold;

[0088] Based on the current detection group, the number of loss function values ​​exceeding the first threshold is summarized and determined, and it is judged that it is greater than a preset second threshold, and the current detection group is determined to be an accuracy abnormality detection group.

[0089] Here, the first threshold is a preset differential threshold, and the second threshold is a preset quantity threshold, which is pre-set according to the network status of the AI ​​computing cluster network and the required computing resources. For different rounds of anomaly detection, the first threshold is set the same, and the second threshold can be set the same or different, which is not limited here.

[0090] Determining that the current detection group is a performance anomaly detection group includes:

[0091] Using the performance value of the current detection group and taking golden as the benchmark, calculate the absolute value of the variance of the current detection group;

[0092] It is determined whether the absolute value of the variance of the current detection group exceeds a preset ratio value of a preset variance reference value. If so, it is determined that the current detection group is a performance abnormality detection group.

[0093] Here, the variance benchmark value and the ratio value are pre-set according to the network status of the AI ​​computing cluster network and the required computing resources. For different rounds of anomaly detection, the variance benchmark value is set the same or different, and the ratio value is set the same or different, which is not limited here.

[0094] The above method is implemented by a detection processing node set in the AI ​​computing power cluster network, in which a computing node anomaly detection platform is deployed to interact with computing nodes in the AI ​​computing power cluster network. Specifically, Figure 2 As shown, Figure 2 A schematic diagram of the structure of a computing node detection device in an AI computing power cluster network provided in an embodiment of the present application, wherein the detection processing node includes: a division unit, a configuration unit, an interaction unit and a detection unit, wherein:

[0095] The division unit is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; under the instruction of the detection unit, the computing nodes in the current detection group are divided into at least one second detection group, the second detection group includes 2 n computing nodes, n is a natural number, and the second detection group is used as the current detection group;

[0096] The configuration unit is used to configure at least one operator in the neural network model to the computing nodes of the current detection group respectively;

[0097] The interaction unit is used to interact with the computing nodes of the current detection group so that the computing nodes of the current detection group train corresponding operators and obtain training results of the current detection group;

[0098] The detection unit is used to determine whether the current detection group is abnormal based on the training results of the current detection group; determine whether the number of computing nodes in the current detection group is 1, and if not, instruct the division unit to divide the current detection group; if yes, determine that the computing nodes in the current detection group are abnormal.

[0099] The detection unit in the above detection processing node, when determining that the computing node in the current detection group is abnormal, further includes:

[0100] The interaction unit is also used to control abnormal computing nodes for fault tolerance, or to control the training process of the neural network model used to complete subsequent AI-related computing tasks based on updating the training strategy of the preset neural network model.

[0101] The present application also provides a detection system for computing nodes in an AI computing power cluster network. Figure 3 As shown, Figure 3 A schematic diagram of the structure of a computing node detection system in an AI computing power cluster network provided in an embodiment of the present application, the system comprising: a detection node and multiple computing nodes, wherein:

[0102] The detection node is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; at least one operator in the neural network model is respectively configured to the computing nodes of the current detection group; interacting with the computing nodes of the current detection group so that the computing nodes of the current detection group train the corresponding operators and obtain the training results of the current detection group; based on the training results of the current detection group, determining that the current detection group is abnormal; determining whether the number of computing nodes in the current detection group is 1, if not, dividing the computing nodes in the current detection group into at least one second detection group, the second detection group including 2 n computing nodes, n is a natural number, the second detection group is used as the current detection group, and the above configuration process is continued; if yes, it is determined that the computing nodes in the current detection group are abnormal;

[0103] The computing node is used to configure the corresponding operator in the neural network model under the configuration of the detection node; under the instruction of the detection node, train the corresponding operator in the neural network model and send the obtained training results to the detection node.

[0104] In the above system, after the detection node determines that the current detection group is abnormal, it is also used to control the abnormal computing node to perform fault tolerance, or based on updating the training strategy of the preset neural network model, control the training process of the neural network model used to complete the subsequent execution of AI-related computing tasks.

[0105] The following is a specific example to illustrate the embodiments of the present application in detail.

[0106] In this example, a real neural network model and training data suitable for the neural network model are used to perform parallel training on the computing nodes in the AI ​​computing power cluster network to obtain training results. The correctness of the computing nodes in executing the training tasks is determined by detecting the training results, thereby identifying abnormal computing nodes.

[0107] In this example, the computing nodes in the AI ​​computing cluster network are GPU servers, each of which has 8 GPU accelerator cards, referred to as accelerator cards.

[0108] Pre-test preparation stage

[0109] In order to facilitate detection, the same neural network model is used for detection in different detection stages in this example. This example can use an existing large language model as a neural network model for detection. In the selection of training data, this example can choose an existing smaller data set. On this basis, this example also introduces a corresponding word segmentation model, which can perform word segmentation on the selected data set.

[0110] The process of the pre-test preparation stage in this example is: the large language model is divided into different operators using PP and TP methods and configured on different computing nodes in the AI ​​computing cluster network, and the selected smaller data set is divided based on DP after word segmentation and provided as training data to the GPU accelerator card in each computing node. Based on this, preparations for testing are made in advance to improve the efficiency of the testing stage.

[0111] Confirm golden

[0112] As a criterion for judging whether the current detection group is abnormal, in another AI computing power cluster network, the other AI computing power cluster network has the same number and type of computing nodes as those in the AI ​​computing power cluster network being detected. After the computing nodes therein are detected and configured using the same detection configuration strategy, the same neural network model and the same training data are used for training to obtain golden, which includes the performance results of the golden parameters, convergence curves, resource consumption and other data.

[0113] Testing process

[0114] In each stage of this example, the DP, PP and TP methods are used to configure the neural network model and training data for detection to each GPU accelerator card in each computing node to be detected.

[0115] The detection process of this example specifically includes three stages:

[0116] 1) Detection phase of 4 GPU servers and 32 GPU accelerator cards

[0117] In this stage, the training data is divided into two groups, the neural network model is divided into four groups by layer, and each layer is divided into four operators, recorded as DP2PP4TP4, which are respectively configured in 32 GPU accelerator cards of four servers.

[0118] The discrete aggregation (Allreduce) communication mode is adopted between GPU servers, the Allreduce communication mode is adopted between GPU accelerator cards in GPU servers, and the point-to-point (P2P) communication mode is adopted between GPU accelerator cards across GPU servers.

[0119] 2) Testing phase of 2 GPU servers and 16 GPU accelerator cards

[0120] In this stage, the training data is not divided. The neural network model is divided into 4 groups by layer, and each layer is divided into 4 operators, denoted as DP1PP4TP4, which are respectively configured in 16 GPU accelerator cards of 2 servers.

[0121] The Allreduce communication method is used between the accelerator cards in the GPU server, and the P2P communication method is used between the GPU accelerator cards across the GPU servers.

[0122] 3) Detection phase of 1 GPU server and 8 GPU accelerator cards

[0123] In this stage, the training data is not divided. The neural network model is divided into two groups by layer, and each layer is divided into four operators, denoted as DP1PP2TP4, which are respectively configured in the eight GPU accelerator cards of one server.

[0124] The GPU accelerator cards in the GPU server use the Allreduce communication method and the P2P communication method.

[0125] Before executing the detection process, in another AI computing power cluster network, the other AI computing power cluster network has the same number and type of computing nodes as those in the AI ​​computing power cluster network being detected. After the computing nodes therein are detected and configured using the same detection configuration strategy, the same neural network model and the same training data are used for training to obtain the golden of the corresponding detection group in the other AI computing power cluster network. The other AI computing power cluster network must meet the following conditions during training: the target loss function value is aligned with the preset loss function standard value, that is, the difference between the target loss function value and the loss function standard value is within the preset deviation value, such as the preset deviation value is 1% of the loss function standard value, and the number of training iterations of the neural network model of the other AI computing power cluster network is the same as the number of training iterations of the neural network model of the test AI computing power cluster network, such as n times, where n is a positive integer.

[0126] Based on the training data obtained from training another AI computing cluster network, taking golden as the benchmark, the absolute value m of the variance of tgs can also be counted.

[0127] According to the above three detection stages, configure each GPU accelerator card in each GPU server of the AI ​​computing cluster network to be tested and perform the test. Figure 4 A schematic diagram of the overall process of detecting a GPU server provided for a specific example of this application. Figure 4 middle:

[0128] Initialize the detection configuration strategy;

[0129] Based on the detection configuration strategy, the GPU server detection group is divided in accordance with the detection stage described in point 1) above, and the neural network model is configured to each GPU accelerator card of the GPU server, and then training is performed, and the output training results are analyzed to determine the abnormal GPU server group;

[0130] For the above abnormal GPU server group, continue to divide it according to the detection stage described in the above point 2), and configure the neural network model to each GPU accelerator card of the divided GPU server group, perform training, analyze and output the training results, analyze the training results, and determine the abnormal GPU server group;

[0131] For the above-mentioned abnormal GPU server group, continue to divide it according to the detection stage described in point 3) above, and configure the neural network model to each GPU accelerator card of the divided GPU server group, conduct training, analyze and output the training results, analyze the training results, identify the abnormal GPU server, and summarize the output.

[0132] Figure 5The flowchart of the method for detecting abnormal GPU servers provided for the specific example of this application includes the following specific steps:

[0133] Step 501: Divide the AI ​​computing cluster network into N detection groups, each detection group has 4 GPU servers, each GPU server has 8 GPU accelerator cards, divide the neural network model into different operators, and configure them on the GPU accelerator cards in the GPU of each detection group;

[0134] Iterate the training of 32 GPU accelerator cards in the four GPU servers according to the number of training iterations to obtain training results, which include loss function values ​​and performance values.

[0135] In this step, the loss function value can be the loss curve, and the performance value is tgs.

[0136] Step 502: For each detection group, for the detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed the preset difference threshold t0; based on the detection group, summarize and determine the number of loss function values ​​that exceed the difference threshold t0, and when it is judged that it is greater than the preset second threshold t1, determine that the detection group is an accuracy abnormality detection group.

[0137] Step 503, for each detection group, the performance value of the detection group is calculated with golden as the benchmark, and the absolute value of the variance m0 of the detection group is calculated; it is determined whether the absolute value of the variance m0 of the detection group exceeds the preset ratio value r0 of the set variance benchmark value v0, and if so, the detection group is determined to be a performance abnormality detection group.

[0138] Step 504: For each detection group, if after the judgment of step 502 and step 503, the detection is not determined as an accuracy abnormality detection group or a performance abnormality detection group, then the GPU servers in the detection group are all normal, and the detection process is exited.

[0139] Step 505: Divide the groups determined as the accuracy anomaly detection group or the performance anomaly detection group in step 502 and step 503 into second detection groups, each of which has two GPU servers, each of which has eight GPU accelerator cards, and divide the neural network model into different operators, which are configured on the GPU accelerator cards in the GPU servers of each detection group;

[0140] The 16 GPU accelerator cards in the two configured GPU servers are iteratively trained according to the number of training iterations to obtain training results, which include loss function values ​​and performance values.

[0141] Step 506: For each second detection group, for the detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed the preset difference threshold t0; based on the detection group, summarize and determine the number of loss function values ​​that exceed the preset difference threshold t0, and when it is judged that it is greater than the preset second threshold t1, determine that the second detection group is an accuracy abnormality detection group.

[0142] Step 507, for each second detection group, calculate the absolute value of the variance m1 of the detection group based on the performance value of the detection group with golden as the benchmark; determine whether the absolute value of the variance m1 of the second detection group exceeds the preset ratio value r1 of the set variance benchmark value v1, and if so, determine that the second detection group is a performance abnormality detection group.

[0143] Step 508: For each second detection group, if after the judgment of step 506 and step 507, the detection group is not determined to be an accuracy abnormality detection group or a performance abnormality detection group, then the GPU servers in the detection group are all normal, and the detection process is exited.

[0144] Step 509: The groups determined as the accuracy anomaly detection group or the performance anomaly detection group in step 506 and step 507 are divided again into third detection groups, each of which has one GPU server, and each GPU server has eight GPU accelerator cards. The neural network model is divided into different operators, which are configured on the GPU accelerator cards in the GPU servers of each third detection group.

[0145] Iterative training is performed on 8 GPU accelerator cards in a configured GPU server according to the number of training iterations to obtain training results, which include loss function values ​​and performance values.

[0146] Step 510: For the third detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed the preset difference threshold t0; determine that the number of loss function values ​​exceeding the preset difference threshold t0 is greater than the preset second threshold t1, and determine that the GPU server is an abnormal accuracy computing node in step 512; otherwise, determine that the GPU server is a normal computing node;

[0147] Step 511, for the performance value of the third detection group, taking golden as the benchmark, calculate the absolute value m3 of the variance of the detection group, and determine whether the absolute value m3 of the variance of the third detection group exceeds the preset ratio value r3 of the set variance benchmark value v3. If so, determine in step 512 that the GPU server is a computing node with abnormal performance. If not, determine that the GPU server is a normal computing node.

[0148] In another embodiment of the present application, an electronic device is also provided, characterized in that it includes an electronic device comprising: a processor; a memory storing a program, wherein the program is configured to implement the computing node detection method in the AI ​​computing power cluster network as described above when executed by the processor.

[0149] In another embodiment of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to execute a computing node detection method in an AI computing power cluster network in the aforementioned embodiment.

[0150] Figure 6 A schematic diagram of an electronic device provided by another embodiment of the present application. Figure 6 As shown, another embodiment of the present application further provides an electronic device, which may include a processor 601, wherein the processor 601 is used to execute the steps of the above-mentioned computing node detection method in an AI computing power cluster network. Figure 6 It can also be seen that the electronic device provided in the above embodiment also includes a non-transitory computer-readable storage medium 602, on which a computer program is stored. When the computer program is executed by the processor 601, the steps of the computing node detection method in the above-mentioned AI computing power cluster network are executed.

[0151] Specifically, the non-transitory computer-readable storage medium 602 can be a general storage medium, such as a mobile disk, a hard disk, FLASH, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), or a portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 602 is executed by the processor 601, it can trigger the processor 601 to execute the various steps of the above-mentioned method for detecting computing nodes in an AI computing power cluster network.

[0152] In practical applications, the non-transitory computer-readable storage medium 602 may be included in the device / apparatus / system described in the above embodiment, or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, each step in the above method for detecting computing nodes in an AI computing power cluster network can be executed.

[0153] Another embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the various steps in the above-mentioned method for detecting computing nodes in an AI computing power cluster network.

[0154] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0156] Specific embodiments are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present application, and any modifications, equivalent substitutions, improvements, etc. made by them should be included in the scope of protection of the present application.

Claims

1. A computing node detection method in an AI computing power cluster network, characterized in that: The method comprises: Divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, and the first detection group is used as the current detection group; configuring at least one operator in the neural network model to the computing nodes of the current detection group respectively, so that the computing nodes of the current detection group train the corresponding operators to obtain the training results of the current detection group; Based on the training result of the current detection group, determining that the current detection group is abnormal; When it is determined that the number of computing nodes in the current detection group is not 1, the computing nodes in the current detection group are divided into at least one second detection group, wherein the second detection group includes 2 n computing nodes, where n is a natural number, taking the second detection group as the current detection group, returning to the step of configuring at least one operator in the neural network model to the computing nodes of the current detection group and subsequent steps, and continuing to execute; When it is determined that the number of computing nodes in the current detection group is 1, it is determined that the computing nodes in the current detection group are abnormal.

2. The method according to claim 1, characterized in that The dividing of computing nodes in the AI ​​computing power cluster network into more than one first detection group includes: Evenly divide the computing nodes in the AI ​​computing power cluster network into the first detection group; The step d of dividing the computing nodes in the current detection group into at least one second detection group includes: The computing nodes in the current detection group are evenly divided into the second detection group.

3. The method according to claim 1, characterized in that The configuring at least one operator in the neural network model to the computing nodes of the current detection group includes: Based on the PP mode of the preset pipeline parallel PP number and the TP mode of the preset tensor parallel TP number, the neural network model is divided into operators, and the divided operators are respectively configured to the computing nodes in the current detection group; In step b, enabling the computing nodes of the current detection group to train corresponding operators includes: Based on the DP method with a preset number of data parallel DPs, the training data used by the neural network model is divided and input in parallel to the corresponding operators of the computing nodes of the current detection group for training.

4. The method according to claim 1 or 3, characterized in that The step of enabling the computing nodes of the current detection group to train the corresponding operators further includes: According to a preset number of training iterations, the corresponding operators are iteratively trained on the computing nodes of the current detection group.

5. The method according to claim 4, characterized in that The computing node of the current detection group is a graphics processing unit (GPU) server, and the GPU server includes at least one GPU accelerator card; The obtaining of the training result of the current detection group includes: The loss function value and / or the performance value, wherein the performance value is the number of tokens executed per second divided by the number of GPU accelerator cards tgs in the computing nodes of the current detection group; Determining that the current detection group is abnormal in step c includes: Comparing the loss function value with a preset loss function standard value to determine that the current detection group is an accuracy abnormality detection group; or / and, The performance value is compared with a preset performance standard value golden to determine that the current detection group is a performance abnormality detection group.

6. The method according to claim 5, characterized in that The determining that the current detection group is an accuracy abnormality detection group comprises: For the current detection group, calculate the difference between the loss function value obtained in each training iteration round and the loss function standard value, and determine the number of loss function values ​​whose calculated difference values ​​exceed a preset first threshold; Based on the current detection group, the number of loss function values ​​exceeding the first threshold is summarized and determined, and it is judged that it is greater than a preset second threshold, and the current detection group is determined to be an accuracy abnormality detection group.

7. The method according to claim 5, characterized in that The determining that the current detection group is a performance abnormality detection group comprises: Using the performance value of the current detection group and taking golden as the benchmark, calculate the absolute value of the variance of the current detection group; It is determined whether the absolute value of the variance of the current detection group exceeds a preset ratio value of a set variance reference value. If so, it is determined that the current detection group is a performance abnormality detection group.

8. A computing node detection device in an AI computing power cluster network, characterized in that: The device comprises: a division unit, a configuration unit, an interaction unit and a detection unit, wherein: The division unit is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; under the instruction of the detection unit, the computing nodes in the current detection group are divided into at least one second detection group, the second detection group includes 2 n computing nodes, n is a natural number, and the second detection group is used as the current detection group; The configuration unit is used to configure at least one operator in the neural network model to the computing nodes of the current detection group respectively; The interaction unit is used to interact with the computing nodes of the current detection group, so that the computing nodes of the current detection group train corresponding operators and obtain training results of the current detection group; The detection unit is used to determine whether the current detection group is abnormal based on the training result of the current detection group; determine whether the number of computing nodes in the current detection group is 1, and if not, instruct the division unit to divide the current detection group; if yes, determine that the computing nodes in the current detection group are abnormal.

9. A computing node detection system in an AI computing power cluster network, characterized in that: The system comprises: a detection node and a plurality of computing nodes, wherein: The detection node is used to divide the computing nodes in the AI ​​computing power cluster network into one or more first detection groups, wherein the first detection group includes 2 n computing nodes, n is a natural number, the first detection group is used as the current detection group; at least one operator in the neural network model is respectively configured to the computing nodes of the current detection group; interacting with the computing nodes of the current detection group so that the computing nodes of the current detection group train the corresponding operators and obtain the training results of the current detection group; based on the training results of the current detection group, determining that the current detection group is abnormal; determining whether the number of computing nodes in the current detection group is 1, if not, dividing the computing nodes in the current detection group into at least one second detection group, the second detection group includes 2 n computing nodes, n is a natural number, taking the second detection group as the current detection group, and continuing to execute the above configuration process; if yes, determining that the computing nodes in the current detection group are abnormal; The computing node is used to configure the corresponding operator in the neural network model under the configuration of the detection node; under the instruction of the detection node, train the corresponding operator in the neural network model and send the obtained training results to the detection node.

10. An electronic device, characterized in that: It includes a processor, wherein the processor executes the computing node detection method in the AI ​​computing power cluster network as described in any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by a processor, the processor executes the computing node detection method in an AI computing power cluster network as described in any one of claims 1 to 7.

Images