Abnormal node determination method and device

Abstract

The embodiment of the invention provides an abnormal node determination method and device. The method comprises the steps that programs running on all nodes in a cluster are determined; one or more node sets are determined in the cluster according to the programs running on the nodes in the cluster, and the programs running on the nodes in the node sets are the same; obtaining log information of each node in the cluster; and for any node set, determining whether an abnormal node exists in the node set according to the log information of each node in the node set, and giving an alarm when the abnormal node exists in the node set. Through the method, the determination efficiency of the abnormal node can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of computing devices, and in particular, to a method and device for determining abnormal nodes. Background Art

[0002] The nodes in the cluster can be used for model training. If a node fails during model training, the model training results may be inaccurate.

[0003] Currently, technicians can analyze the logs generated by nodes during model training to find abnormal nodes with abnormal operation before the nodes fail, and can process the abnormal nodes to reduce the probability of node failure. However, the above method has a low efficiency in determining abnormal nodes. Summary of the invention

[0004] The embodiments of the present application provide a method and device for determining an abnormal node, which are used to solve the technical problem of low efficiency in determining abnormal nodes.

[0005] In a first aspect, an embodiment of the present application provides a method for determining an abnormal node, including:

[0006] Determine the programs running on each node in the cluster;

[0007] Determine one or more node sets in the cluster according to the program running on each node in the cluster, wherein the program running on each node in the node set is the same;

[0008] Obtaining log information of each node in the cluster;

[0009] For any node set, determine whether there is an abnormal node in the node set according to the log information of each node in the node set, and issue an alarm when the abnormal node exists in the node set.

[0010] In the above scheme, the electronic device can determine the program running on each node in the cluster; can determine one or more node sets in the cluster based on the program running on each node in the cluster; can obtain the log information of each node in the cluster; for any node set, can determine whether there are abnormal nodes in the node set based on the log information of each node in the node set, and issue an alarm when there are abnormal nodes in the node set. In the above method, the electronic device can determine whether there are abnormal nodes in the node set based on the log information of each node in the node set, thereby improving the efficiency of determining abnormal nodes.

[0011] In a possible implementation, determining, for any one of the node sets, whether there is an abnormal node in the node set according to the log information of each node in the node set includes:

[0012] Determine a plurality of indicator values ​​corresponding to the node set according to the log information of each node in the node set;

[0013] Determine whether there is an abnormal index value among the multiple index values ​​corresponding to the node set;

[0014] If the abnormal index value does not exist in the multiple index values ​​corresponding to the node set, determining that the abnormal node does not exist in the node set;

[0015] If the abnormal index value exists in the multiple index values ​​corresponding to the node set, it is determined that there is an abnormal node in the node set, and the node corresponding to the abnormal index value is determined as the abnormal node.

[0016] In the above solution, the electronic device can determine whether there is an abnormal node in the node according to the multiple indicator values ​​corresponding to the node set, thereby achieving the purpose of determining the abnormal node.

[0017] In a possible implementation, determining a plurality of indicator values ​​corresponding to the node set according to the log information of each node in the node set includes:

[0018] For any node in the node set, determine at least one indicator value of the node according to the log information of the node;

[0019] The multiple index values ​​corresponding to the node set include: the at least one index value of each node in the node set.

[0020] In the above solution, the electronic device can determine the index value of the node according to the log information of the node, thereby achieving the purpose of determining the index value of the node.

[0021] In a possible implementation manner, determining at least one indicator value of the node according to the log information of the node includes:

[0022] Determining at least one indicator field to be monitored;

[0023] For any indicator field, obtaining at least one operating parameter corresponding to the indicator field from the log information of the node, and determining the indicator value corresponding to the indicator field according to the at least one operating parameter corresponding to the indicator field;

[0024] The at least one indicator value of the node includes: an indicator value corresponding to the at least one indicator field determined according to the operating parameters of the node.

[0025] In the above solution, the electronic device can calculate the indicator value corresponding to the indicator field according to at least one operating parameter corresponding to the indicator field, thereby achieving the purpose of determining the indicator value corresponding to the indicator field.

[0026] In a possible implementation manner, determining whether there is an abnormal index value among the multiple index values ​​corresponding to the node set includes:

[0027] According to at least one indicator field to be monitored, determining at least one indicator group from a plurality of indicator values ​​corresponding to the node set, the indicator group including indicator values ​​of each node in the node set corresponding to the same indicator field;

[0028] For any one of the indicator groups, determining whether the indicator group includes an abnormal indicator value according to the indicator value included in the indicator group;

[0029] If the abnormal index value is not included in the index group, determining that the abnormal index value does not exist in the multiple index values ​​corresponding to the node set;

[0030] If there is an abnormal index value included in the index group, it is determined that the abnormal index value exists in the multiple index values ​​corresponding to the node set.

[0031] In the above scheme, the electronic device can determine the indicator values ​​corresponding to the same indicator field among the multiple indicator values ​​corresponding to the node set as an indicator group, and can determine whether there are abnormal indicator values ​​in the node set based on whether there are abnormal indicator values ​​in the indicator group, thereby achieving the purpose of determining whether there are abnormal indicator values ​​among the multiple indicator values ​​corresponding to the node set.

[0032] In a possible implementation, for any one of the indicator groups, determining whether the indicator group includes an abnormal indicator value according to a plurality of indicator values ​​included in the indicator group includes:

[0033] Determining the mean and variance of a plurality of indicator values ​​included in the indicator group;

[0034] Determine the alarm range corresponding to the indicator group according to the mean and the variance;

[0035] According to the multiple indicator values ​​included in the indicator group and the alarm range, it is determined whether the indicator group includes an abnormal indicator value.

[0036] In the above scheme, the electronic device can determine whether the indicator group includes abnormal indicator values ​​according to the alarm range corresponding to the indicator group and the indicator values ​​in the indicator group, thereby achieving the purpose of determining whether the indicator group includes abnormal indicator values.

[0037] In a possible implementation, determining whether the indicator group includes an abnormal indicator value according to the multiple indicator values ​​included in the indicator group and the alarm range includes:

[0038] Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group;

[0039] If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value;

[0040] If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group includes the abnormal indicator value, and the target indicator value is determined as the abnormal indicator value.

[0041] In the above solution, the electronic device can determine the target index value outside the warning range as an abnormal index value, thereby achieving the purpose of determining the abnormal index value.

[0042] In a possible implementation, determining whether the indicator group includes an abnormal indicator value according to the multiple indicator values ​​included in the indicator group and the alarm range includes:

[0043] Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group;

[0044] If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value;

[0045] If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined whether the indicator group includes an abnormal indicator value according to the target indicator value.

[0046] In the above solution, the electronic device can determine whether the indicator group includes an abnormal indicator value according to the target indicator value, thereby achieving the purpose of determining the abnormal indicator value.

[0047] In a possible implementation, determining whether the indicator group includes an abnormal indicator value according to the target indicator value includes:

[0048] Determine a preset number of historical indicator values ​​corresponding to the target indicator value, and determine whether each historical indicator value is within a corresponding alarm range;

[0049] If the preset number of historical indicator values ​​are all outside the corresponding alarm range, determining that the indicator group includes the abnormal indicator value, and determining that the target indicator value is the abnormal indicator value;

[0050] If there is a historical indicator value in the preset number of historical indicator values ​​that is within the corresponding alarm range, it is determined that the indicator group does not include the abnormal indicator value.

[0051] In the above solution, the electronic device can determine whether the indicator group includes an abnormal indicator value based on the target indicator value and the historical indicator value corresponding to the target indicator value, thereby achieving the purpose of determining the abnormal indicator value.

[0052] In a possible implementation, determining the alarm range corresponding to the indicator group according to the mean and the variance includes:

[0053] Determine the product of the variance and a preset value;

[0054] Determine the difference between the mean and the product as the minimum value of the alarm range;

[0055] The sum of the mean value and the product is determined as the maximum value of the alarm range.

[0056] In the above solution, the electronic device can determine the alarm range corresponding to the indicator group according to the variance corresponding to the indicator group, thereby achieving the purpose of determining the alarm range corresponding to the indicator group.

[0057] In a first aspect, an embodiment of the present application provides a device for determining an abnormal node, the device for determining an abnormal node comprising: a determination module and an acquisition module, wherein:

[0058] The determination module is used to determine the program running on each node in the cluster;

[0059] The determination module is further used to determine one or more node sets in the cluster according to the programs running on each node in the cluster, wherein the programs running on each node in the node set are the same;

[0060] The acquisition module is used to obtain log information of each node in the cluster;

[0061] The determination module is further configured to determine, for any one of the node sets, whether there is an abnormal node in the node set according to the log information of each node in the node set, and to issue an alarm when the abnormal node exists in the node set.

[0062] In the above scheme, the abnormal node determination device can determine the program running on each node in the cluster; can determine one or more node sets in the cluster based on the program running on each node in the cluster; can obtain the log information of each node in the cluster; for any node set, can determine whether there are abnormal nodes in the node set based on the log information of each node in the node set, and issue an alarm when there are abnormal nodes in the node set. In the above method, the abnormal node determination device can determine whether there are abnormal nodes in the node set based on the log information of each node in the node set, thereby improving the efficiency of determining abnormal nodes.

[0063] In a possible implementation, the determining module is specifically configured to:

[0064] Determine a plurality of indicator values ​​corresponding to the node set according to the log information of each node in the node set;

[0065] Determine whether there is an abnormal index value among the multiple index values ​​corresponding to the node set;

[0066] If the abnormal index value does not exist in the multiple index values ​​corresponding to the node set, determining that the abnormal node does not exist in the node set;

[0067] If the abnormal index value exists in the multiple index values ​​corresponding to the node set, it is determined that there is an abnormal node in the node set, and the node corresponding to the abnormal index value is determined as the abnormal node.

[0068] In the above solution, the determination module can determine whether there is an abnormal node in the node according to multiple indicator values ​​corresponding to the node set, thereby achieving the purpose of determining the abnormal node.

[0069] In a possible implementation, the determining module is specifically configured to:

[0070] For any node in the node set, determine at least one indicator value of the node according to the log information of the node;

[0071] The multiple index values ​​corresponding to the node set include: the at least one index value of each node in the node set.

[0072] In the above solution, the determination module can determine the index value of the node according to the log information of the node, thereby achieving the purpose of determining the index value of the node.

[0073] In a possible implementation, the determining module is specifically configured to:

[0074] Determining at least one indicator field to be monitored;

[0075] For any indicator field, obtaining at least one operating parameter corresponding to the indicator field from the log information of the node, and determining the indicator value corresponding to the indicator field according to the at least one operating parameter corresponding to the indicator field;

[0076] The at least one indicator value of the node includes: an indicator value corresponding to the at least one indicator field determined according to the operating parameters of the node.

[0077] In the above solution, the determination module can calculate the indicator value corresponding to the indicator field according to at least one operating parameter corresponding to the indicator field, thereby achieving the purpose of determining the indicator value corresponding to the indicator field.

[0078] In a possible implementation, the determining module is specifically configured to:

[0079] According to at least one indicator field to be monitored, determining at least one indicator group from a plurality of indicator values ​​corresponding to the node set, the indicator group including indicator values ​​of each node in the node set corresponding to the same indicator field;

[0080] For any one of the indicator groups, determining whether the indicator group includes an abnormal indicator value according to the indicator value included in the indicator group;

[0081] If the abnormal index value is not included in the index group, determining that the abnormal index value does not exist in the multiple index values ​​corresponding to the node set;

[0082] If there is an abnormal index value included in the index group, it is determined that the abnormal index value exists in the multiple index values ​​corresponding to the node set.

[0083] In the above scheme, the determination module can determine the indicator values ​​corresponding to the same indicator field among the multiple indicator values ​​corresponding to the node set as an indicator group, and can determine whether there are abnormal indicator values ​​in the node set based on whether there are abnormal indicator values ​​in the indicator group, thereby achieving the purpose of determining whether there are abnormal indicator values ​​among the multiple indicator values ​​corresponding to the node set.

[0084] In a possible implementation, the determining module is specifically configured to:

[0085] Determining the mean and variance of a plurality of indicator values ​​included in the indicator group;

[0086] Determine the alarm range corresponding to the indicator group according to the mean and the variance;

[0087] According to the multiple indicator values ​​included in the indicator group and the alarm range, it is determined whether the indicator group includes an abnormal indicator value.

[0088] In the above scheme, the determination module can determine whether the indicator group includes abnormal indicator values ​​according to the alarm range corresponding to the indicator group and the indicator value in the indicator group, thereby achieving the purpose of determining whether the indicator group includes abnormal indicator values.

[0089] In a possible implementation, the determining module is specifically configured to:

[0090] Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group;

[0091] If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value;

[0092] If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group includes the abnormal indicator value, and the target indicator value is determined as the abnormal indicator value.

[0093] In the above solution, the determination module can determine the target indicator value outside the alarm range as an abnormal indicator value, thereby achieving the purpose of determining the abnormal indicator value.

[0094] In a possible implementation, the determining module is specifically configured to:

[0095] Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group;

[0096] If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value;

[0097] If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined whether the indicator group includes an abnormal indicator value according to the target indicator value.

[0098] In the above solution, the determination module can determine whether the indicator group includes an abnormal indicator value according to the target indicator value, thereby achieving the purpose of determining the abnormal indicator value.

[0099] In a possible implementation, the determining module is specifically configured to:

[0100] Determine a preset number of historical indicator values ​​corresponding to the target indicator value, and determine whether each historical indicator value is within a corresponding alarm range;

[0101] If the preset number of historical indicator values ​​are all outside the corresponding alarm range, determining that the indicator group includes the abnormal indicator value, and determining that the target indicator value is the abnormal indicator value;

[0102] If there is a historical indicator value in the preset number of historical indicator values ​​that is within the corresponding alarm range, it is determined that the indicator group does not include the abnormal indicator value.

[0103] In the above solution, the determination module can determine whether the indicator group includes an abnormal indicator value based on the target indicator value and the historical indicator value corresponding to the target indicator value, thereby achieving the purpose of determining the abnormal indicator value.

[0104] In a possible implementation, the determining module is specifically configured to:

[0105] Determine the product of the variance and a preset value;

[0106] Determine the difference between the mean and the product as the minimum value of the alarm range;

[0107] The sum of the mean value and the product is determined as the maximum value of the alarm range.

[0108] In the above scheme, the determination module can determine the alarm range corresponding to the indicator group according to the variance corresponding to the indicator group, thereby achieving the purpose of determining the alarm range corresponding to the indicator group.

[0109] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

[0110] The memory stores computer-executable instructions;

[0111] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method described in any one of the first aspects.

[0112] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any one of the first aspects.

[0113] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method shown in any one of the first aspects when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0115] Figure 1A data parallel schematic diagram provided for an embodiment of the present application;

[0116] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0117] Figure 3 A cluster diagram provided for an embodiment of the present application;

[0118] Figure 4 A schematic diagram of a method for determining an abnormal node provided in an embodiment of the present application;

[0119] Figure 5 A flowchart of a method for determining an abnormal node provided in an embodiment of the present application;

[0120] Figure 6 A schematic diagram of program information acquisition provided in an embodiment of the present application;

[0121] Figure 7 A log information acquisition schematic diagram provided in an embodiment of the present application;

[0122] Figure 8 A flowchart of another abnormal node determination method provided in an embodiment of the present application;

[0123] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0124] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0125] To facilitate understanding, the technical terms involved in the embodiments of the present application are first explained.

[0126] Distributed training: refers to using multiple machines to process training data in parallel, thereby accelerating the training process of deep learning models.

[0127] Data parallelism: is a technique that divides data into small chunks and processes these chunks simultaneously on multiple processors. Figure 1 Data parallelism is explained.

[0128] Figure 1 A data parallel diagram provided for an embodiment of the present application. Figure 1, the cluster includes node 0, node 1 and node 2. Assume that the same model to be trained is deployed on node 0, node 1 and node 2. Assume that the training data can be divided into training data 0, training data 1 and training data 2. Figure 1 As shown, node 0 can receive training data 0 for model training, node 1 can receive training data 1 for model training, and node 2 can receive training data 2 for model training. The above model training method can be called a data parallel model training method.

[0129] Pipeline parallelism: For a model, each layer of the model is deployed on different processors and executed simultaneously.

[0130] Tensor parallelism: For each layer of the model, the parameters of each layer are split vertically and deployed on different processors or computing devices.

[0131] For ease of understanding, the following Figure 2 , the application scenarios involved in the embodiments of the present application are described.

[0132] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application. Figure 2 , a cluster may include multiple nodes (e.g., node 0, node 1, node 2, ..., and node n). The nodes in the cluster may communicate with each other (not shown in the figure). For example, the cluster may be an artificial intelligence (AI) cluster.

[0133] The node may be a processor that can perform computing tasks or graphics processing, for example, a central processing unit (CPU) or a graphics processing unit (GPU).

[0134] The multiple nodes in the above cluster can be set in one or more electronic devices. That is, for any two nodes, the two nodes can be set in the same electronic device, or the two nodes can also be set in different electronic devices. Among them, the electronic device can be a server, a laptop computer, a desktop computer, a tablet computer, etc. The server can be, for example, a GPU server, an AI server, or a general server, etc.

[0135] The above cluster can be used for model training. During the model training process, the nodes in the cluster need to complete a large number of computing tasks, and the model training process takes a long time. This makes it easy for faults to occur during the model training process.

[0136] For example, during model training, hardware failures may occur due to node overheating, unstable power supply, or physical damage, and software failures may occur due to logical errors in the code.

[0137] On the one hand, failures will interrupt the model training process and increase the time cost of model training; on the other hand, failures may cause inaccurate training results, resulting in poor model training effects.

[0138] To reduce the probability of failure, abnormal nodes with abnormal operation can be checked during the model training process and processed. In this way, the probability of failure caused by abnormal nodes during the model training process can be reduced, thereby reducing the probability of failure during the model training process.

[0139] An embodiment of the present application proposes a method for determining an abnormal node.

[0140] Next, combine Figure 3 , the cluster involved in the embodiment of the present application is described.

[0141] Figure 3 A cluster diagram provided for an embodiment of the present application. Figure 2 , the cluster may include node 0, node 1, node 2, ... and node 11.

[0142] Multiple nodes in a cluster can be divided into multiple node groups, each node group includes the same number of nodes. The multiple can be two or more. For example, Figure 2 The cluster shown may be divided into node group A and node group B, node group A may include node 0, node 1, node 2, ... and node 5, and node group B may include node 6, node 7, node 8, ... and node 11.

[0143] The same model to be trained can be deployed in each node group in the cluster, that is, the model to be trained deployed in each node group is the same. During the model training process, each node group can receive different training data, can perform model training based on the received training data, and can inform each other of their own training results. This model training method can also be called a data parallel model training method. The data parallel model training method can speed up the model training speed. For example, Figure 2 In the cluster shown, the model W to be trained can be deployed on node group A, and the model W to be trained can also be deployed on node group B.

[0144] The model to be trained can generally be composed of multiple layers, and each layer can be used to perform a specific computing task. For example, the model to be trained can include an input layer and a fully connected layer. The nodes in each node group can be divided into multiple sub-node groups. The number of sub-node groups can be the same as the number of layers of the model to be trained. One layer of the model to be trained can be deployed in each sub-node group, that is, the nodes in each sub-node group can be used to train one layer of the model to be trained. This model training method is also called a pipeline parallel model training method.

[0145] For example, Figure 2 In the cluster shown, it is assumed that the model W to be trained deployed on node group A consists of 3 layers. Multiple nodes in node group A can be divided into 3 sub-node groups, which can be sub-node group ①, sub-node group ② and sub-node group ③ respectively. The first layer of the model W to be trained can be deployed in sub-node group ①, the second layer of the model W to be trained can be deployed in sub-node group ②, and the third layer of the model W to be trained can be deployed in sub-node group ③. Similarly, multiple nodes in node group B can also be divided into 3 sub-node groups, which can be sub-node group ④, sub-node group ⑤ and sub-node group ⑥ respectively. The first layer of the model W to be trained can be deployed in sub-node group ④, the second layer of the model W to be trained can be deployed in sub-node group ⑤, and the third layer of the model W to be trained can be deployed in sub-node group ⑥.

[0146] In order to reduce the processing load of a single node, a layer of the model to be trained can generally be divided into multiple parts, and one part can be deployed on one node. This model training method is also called a tensor parallel model training method. In the specific implementation process, a layer of the model to be trained can be divided into multiple parts according to the number of nodes included in the sub-node group.

[0147] For example, Figure 2In the cluster shown, it is assumed that the first layer of the model W to be trained is deployed in subnode group ① and subnode group ④, the second layer of the model W to be trained is deployed in subnode group ② and subnode group ⑤, and the third layer of the model W to be trained is deployed in subnode group ③ and subnode group ⑥. Then the first layer of the model W to be trained can be divided into two parts, and the first part of the first layer can be deployed on node 0, and the second part of the first layer can be deployed on node 1; the second layer of the model W to be trained can be divided into two parts, and the first part of the second layer can be deployed on node 2, and the second part of the second layer can be deployed on node 3; the third layer of the model W to be trained can be divided into two parts, and the first part of the third layer can be deployed on node 4, and the second part of the third layer can be deployed on node 5. Similarly, the first part of the first layer can be deployed on node 6, and the second part of the first layer can be deployed on node 7, the first part of the second layer can be deployed on node 8, and the second part of the second layer can be deployed on node 9, the first part of the third layer can be deployed on node 10, and the second part of the third layer can be deployed on node 11.

[0148] In summary, Figure 2 In the cluster shown, the same part of the model W to be trained can be deployed on node 0 and node 6, the same part of the model W to be trained can be deployed on node 1 and node 7, ..., the same part of the model W to be trained can be deployed on node 5 and node 11. In other words, the program code running on node 0 and node 6 can be the same, the program code running on node 1 and node 7 can be the same, ..., the program code running on node 5 and node 11 can be the same.

[0149] In the abnormal node determination method provided by the embodiment of the present application, the log information of each node in the cluster can be obtained, and the log information of multiple nodes running the same program code can be used to determine whether there are abnormal nodes in the multiple nodes. In this way, for multiple nodes running the same program code, it is possible to determine whether there are abnormal nodes in the multiple nodes by analyzing the log information of the multiple nodes. In the above method, there is no need to analyze the log information of all nodes in the set, so that the processing amount of log information is small, and the efficiency of determining abnormal nodes is improved.

[0150] In addition, in the abnormal node determination method provided in the embodiment of the present application, abnormal nodes can be determined in a timely manner, thereby reducing the impact of abnormal nodes on the model training progress and saving the time cost of model training.

[0151] In addition, in the abnormal node determination method provided in the embodiment of the present application, the abnormal node can be determined based on the log information of multiple nodes running the same program code, so that the accuracy of determining the abnormal node is high. In addition, the above method does not require manual analysis and processing of log information, avoids manual omissions, and further improves the accuracy of determining the abnormal node.

[0152] Next, combine Figure 4 , the abnormal node determination method provided in the embodiment of the present application is described.

[0153] Figure 4 A schematic diagram of a method for determining abnormal nodes provided in an embodiment of the present application. Figure 4 , the cluster may include node 0, node 1, node 2, ... and node n, where n is an integer greater than or equal to 1. The electronic device may communicate with any node in the cluster.

[0154] The electronic device can obtain program information and log information of each node from the cluster.

[0155] The program information of a node may be information used to characterize the program code running on the node. For example, the program information may be an identifier of the program code, or a field used to represent the program code.

[0156] The log information of a node may be information recording the operation status of the node during the model training process. For example, the log information of a node may include the memory usage and processor utilization of the node.

[0157] The electronic device can divide multiple nodes running the same program code in the cluster into a node set according to the program information of each node, and can determine whether there are abnormal nodes in the node set according to the log information of each node in the node set. If there is an abnormal node in any node set in the cluster, the electronic device can issue an alarm.

[0158] The electronic device may be an electronic device with processing capability, information receiving and sending capability and / or display capability. When issuing an alarm, the electronic device may send alarm information to the user terminal, or may display the alarm information, which is not limited in this embodiment.

[0159] It should be noted that the electronic device may be an electronic device in the cluster that does not participate in model training, or the electronic device may be an electronic device outside the cluster. Figure 4 The electronic device being an electronic device outside the cluster is merely used as an example for description, which does not constitute a limitation on the technical solution provided in the embodiment of the present application.

[0160] The technical solutions of the embodiments of the present application are described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0161] Figure 5 The flowchart of an abnormal node determination method provided in an embodiment of the present application is shown in FIG. The execution subject of the method may be an electronic device, or an abnormal node determination device set in the electronic device. The electronic device may be, for example, Figure 4 The electronic device shown. The abnormal node determination device can be implemented by software or by a combination of software and hardware. For ease of understanding, the following description is made by taking the execution subject as an electronic device as an example. Figure 5 , the method may include:

[0162] S501: Determine the program running on each node in the cluster.

[0163] Each node in the cluster may be a node used for model training in the cluster.

[0164] The program running on the node may be the program code running on the node during model training.

[0165] It should be understood that, for any two nodes in the cluster, if the program codes running on the two nodes are the same, then the model training tasks performed by the two nodes are the same.

[0166] In this embodiment, the electronic device can obtain program information of each node in the cluster, and can determine the program running on each node in the cluster according to the program information of each node in the cluster.

[0167] Next, combine Figure 6 , a method for an electronic device to obtain program information of each node in a cluster is described.

[0168] Figure 6 A schematic diagram of program information acquisition provided in an embodiment of the present application. Figure 6 , a cluster may include multiple nodes, and the multiple nodes may be node 0, node 1, node 2, ... and node n. A software development kit (SDK) may be deployed in each node.

[0169] SDK can be used to collect node program information.

[0170] The electronic device can communicate with each node in the cluster. Furthermore, the electronic device can call the SDK deployed on each node in the cluster.

[0171] In the actual implementation process, the SDK deployed on each node can collect the program information of the corresponding node. The electronic device can call the SDK deployed on each node in the cluster to obtain the program information of each node through the SDK. Figure 6 As shown, the electronic device can obtain the program information of node 0 through the SDK deployed on node 0, can obtain the program information of node 1 through the SDK deployed on node 1, can obtain the program information of node 2 through the SDK deployed on node 2, ..., and can obtain the program information of node n through the SDK deployed on node n.

[0172] S502: Determine one or more node sets in the cluster according to the programs running on each node in the cluster.

[0173] The same program runs on each node in the node set.

[0174] In this embodiment, the electronic device may determine multiple nodes in the cluster that run the same program as a node set.

[0175] For example, Figure 3 For example, electronic devices can Figure 3 The nodes 0 and 6 shown are determined as a node set, and the nodes 1 and 7 can be determined as a node set, and so on.

[0176] S503: Obtain log information of each node in the cluster.

[0177] In this embodiment, when the model is trained by the cluster, the model deployed in each node group in the cluster can be trained multiple times with a fixed number of sample data. For example, the fixed number can be 50, and the model deployed in each node group in the cluster can be trained with 50 sample data each time.

[0178] For any node in the cluster, the log information of the node may be information recording the operation status of the node during a training process. For example, the log information of the node may be the processor utilization rate of the node during a training process.

[0179] Next, combine Figure 7 , a method for an electronic device to obtain log information of each node in a cluster is described.

[0180] Figure 7 A log information acquisition diagram provided in the embodiment of the present application. Figure 7 ,exist Figure 6 On this basis, a performance collection module can also be deployed on each node in the cluster.

[0181] The performance collection module may be a software module for collecting log information of a node. Specifically, the performance collection module may obtain a training trace file (Trace File) of the node, and may collect log information of the node from the training trace file.

[0182] The electronic device can communicate with the performance collection module of each node in the cluster.

[0183] In the actual implementation process, the performance collection module deployed on each node can collect the log information of the corresponding node. The electronic device can obtain the log information of each node through the performance collection module. Figure 7 As shown, the electronic device can obtain the log information of node 0 through the performance collection module deployed on node 0, can obtain the log information of node 1 through the performance collection module deployed on node 1, can obtain the log information of node 2 through the performance collection module deployed on node 2, ..., and can obtain the log information of node n through the performance collection module deployed on node n.

[0184] S504: for any node set, determine whether there is an abnormal node in the node set according to the log information of each node in the node set, and issue an alarm when there is an abnormal node in the node set.

[0185] It should be noted that the method for the electronic device to determine whether there is an abnormal node in each node set is the same. In the embodiment of the present application, taking any node set as an example, the method for determining whether there is an abnormal node in the node set is described.

[0186] An abnormal node may be a node with an abnormal indicator value during the model training process.

[0187] The indicator value may be a value that measures the performance and health of the node. For example, the indicator value may be a value corresponding to the processor usage rate and the memory usage rate of the node.

[0188] In this embodiment, the electronic device can determine multiple indicator values ​​corresponding to the node set based on the log information of each node in the node set; determine whether there is an abnormal indicator value among the multiple indicator values ​​corresponding to the node set; if there is no abnormal indicator value among the multiple indicator values ​​corresponding to the node set, determine that there is no abnormal node in the node set; if there is an abnormal indicator value among the multiple indicator values ​​corresponding to the node set, determine that there is an abnormal node in the node set, and determine the node corresponding to the abnormal indicator value as an abnormal node.

[0189] The multiple index values ​​corresponding to the node set may be the index values ​​of the multiple nodes included in the node set. For example, assuming that the node set includes node 0 and node 6, the multiple index values ​​corresponding to the node set may be the index value of node 0 and the index value of node 6.

[0190] Each node can have at least one metric value.

[0191] Exemplarily, assuming that the node set includes node 0 and node 6, the multiple index values ​​corresponding to the node set include index value 1 of node 0, index value 2 of node 0, index value 1 of node 6, and index value 2 of node 6. If the electronic device determines that there is no abnormal index value among index value 1 of node 0, index value 2 of node 0, index value 1 of node 6, and index value 2 of node 6, then there is no abnormal node in the node set. If the electronic device determines that index value 1 of node 6 is an abnormal index value, node 6 can be determined as an abnormal node.

[0192] It should be noted that the method for the electronic device to determine whether there is an abnormal index value among the multiple index values ​​corresponding to the node set can refer to Figure 8 The embodiments are not described in detail here.

[0193] In this embodiment, when the electronic device issues an alarm, the alarm information may be sent to the user terminal, or the alarm information may be displayed, which is not limited in this embodiment.

[0194] The alarm information may include the flag of the abnormal node and the abnormal index value of the abnormal node.

[0195] In the abnormal node determination method provided in this embodiment, the electronic device can determine the program running on each node in the cluster; can determine one or more node sets in the cluster based on the program running on each node in the cluster; can obtain the log information of each node in the cluster; for any node set, can determine whether there is an abnormal node in the node set based on the log information of each node in the node set, and issue an alarm when there is an abnormal node in the node set. In the above method, the electronic device can determine whether there is an abnormal node in the node set based on the log information of each node in the node set, thereby improving the efficiency of determining the abnormal node.

[0196] Based on the above embodiments, Figure 8 , a method for an electronic device to determine whether there is an abnormal indicator value among multiple indicator values ​​corresponding to a node set, and further illustrate the abnormal node determination method provided in an embodiment of the present application.

[0197] Figure 8 A flowchart of another abnormal node determination method provided in an embodiment of the present application. The execution subject of the method may be an electronic device, or an abnormal node determination device set in the electronic device. The electronic device may be, for example, Figure 4The electronic device shown. The abnormal node determination device can be implemented by software or by a combination of software and hardware. For ease of understanding, the following description is made by taking the execution subject as an electronic device as an example. Figure 8 , the method may include:

[0198] S801. Determine the program running on each node in the cluster.

[0199] S802: Determine one or more node sets in the cluster according to the programs running on each node in the cluster.

[0200] S803: Obtain log information of each node in the cluster.

[0201] It should be noted that the specific implementation method of S801-S803 can refer to S501-S503 and will not be repeated here.

[0202] S804: For any node set, determine multiple indicator values ​​corresponding to the node set according to log information of each node in the node set.

[0203] It should be noted that the method for the electronic device to determine the multiple index values ​​corresponding to each node set is the same. In the embodiment of the present application, taking any node set as an example, the method for the electronic device to determine the multiple index values ​​corresponding to the node set is described.

[0204] In this embodiment, for any node in the node set, the electronic device can determine at least one indicator value of the node based on the log information of the node; wherein the multiple indicator values ​​corresponding to the node set include: at least one indicator value of each node in the node set.

[0205] Specifically, the electronic device may determine at least one index value of the corresponding node according to the log information. The multiple index values ​​corresponding to the node set may include at least one index value of each node in the node set.

[0206] It should be noted that the at least one mentioned in this embodiment may be one or more, and the multiple may be two or more.

[0207] For any node in the node set, when the electronic device determines at least one indicator value of the node based on the log information of the node, it can determine at least one indicator field to be monitored; for any indicator field, obtain at least one operating parameter corresponding to the indicator field from the log information of the node, and determine the indicator value corresponding to the indicator field based on the at least one operating parameter corresponding to the indicator field; wherein, the at least one indicator value of the node includes: the indicator value corresponding to at least one indicator field determined based on the operating parameters of the node.

[0208] The at least one indicator field to be monitored is a indicator field that is pre-set in the electronic device and needs to be monitored. For example, the indicator field may be a processor usage rate and a memory usage rate.

[0209] In the abnormal node determination method provided in the embodiment of the present application, there is no limitation on the at least one indicator field to be monitored, as long as it is possible to determine whether there is an abnormal node in the cluster based on the indicator value corresponding to the at least one indicator field.

[0210] Optionally, at least one indicator field may include the number of executions of a key operator in a node, the duration of each execution of the key operator, etc. The instruction throughput and instruction execution capability of each node in the cluster may be determined through the above indicator fields.

[0211] Optionally, at least one indicator field may include the memory usage of the node, the memory usage of key operators in the node, etc. The cache performance, memory management efficiency, and task processing capability of each node in the cluster can be determined through the above indicator fields.

[0212] Optionally, at least one indicator field may include the communication duration of the node, the computational communication ratio of the node, and the processor utilization of the node, etc. The communication efficiency and delay performance of each node in the cluster may be determined through the above indicator fields.

[0213] The key operator may be an operator that is more important in the model training process. In the embodiment of the present application, the key operator may be predetermined according to actual needs. Optionally, the key operator may include a graphics processor kernel (GPU kernel) operator, a data copy operator, a communication operator, and a data loading operator.

[0214] The memory usage of a node can be the maximum memory usage of the node during a training process.

[0215] For any key operator, the memory usage of the key operator in the node can be the maximum memory usage of the key operator in a training process.

[0216] The communication duration of a node may be the duration for which the node communicates with other nodes during a training process.

[0217] The computation-communication ratio of a node may be the ratio of the duration of simultaneous computation of the node in the process of communicating with other nodes to the duration of communication of the node with other nodes during one training process.

[0218] The processor utilization of the node may be CPU utilization, GPU utilization, or neural processing unit (NPU) utilization, etc. If the node is a CPU, the processor utilization may be CPU utilization; if the node is a GPU, the processor utilization may be GPU utilization; if the node is an NPU, the processor utilization may be NPU utilization.

[0219] The at least one operating parameter corresponding to the indicator field may be an operating parameter included in the log information and used to calculate the indicator value corresponding to the indicator field. In a specific implementation, the indicator value corresponding to the indicator field may be determined according to the at least one operating parameter corresponding to the indicator field.

[0220] It should be noted that, for the indicator value corresponding to any indicator field, the electronic device stores a method for determining the indicator value according to the operating parameters corresponding to the indicator field. In the specific implementation process, the electronic device can determine the indicator value corresponding to the indicator field according to the corresponding method and at least one operating parameter.

[0221] In this embodiment, there are at least two situations in which the indicator value corresponding to the indicator field is determined according to at least one operating parameter corresponding to the indicator field:

[0222] Case 1: An operating parameter corresponding to the indicator field is determined as the indicator value corresponding to the indicator field.

[0223] In this case, the number of operating parameters corresponding to the indicator field may be 1, and the indicator value corresponding to the indicator field may be equal to the operating parameter corresponding to the indicator field. The electronic device may directly obtain the operating parameter corresponding to the indicator field from the log information according to the indicator field, and may determine the operating parameter corresponding to the indicator field as the indicator value corresponding to the indicator field, so that the determination efficiency of the indicator value is high.

[0224] In this embodiment, the operating parameters that may be included in the log information may include: the number of times each key operator in the node performs calculations, the duration of each key operator performing calculations each time, the maximum memory usage of the node, the maximum memory usage of each key operator, the duration of the node communicating with other nodes, the duration of the node both communicating with other nodes and performing calculations, and the processor utilization of the node, etc.

[0225] In this embodiment, for any key operator, the electronic device can determine the number of times the key operator in the log information executes calculations as the number of executions of the key operator; the duration used by the key operator in the log information for each execution of calculations can be determined as the duration occupied by the key operator each time it is executed; and the maximum memory usage of the key operator in the log information can be determined as the memory usage of the key operator. The number of times the key operator executes calculations, the duration used by the key operator each time it executes calculations, and the maximum memory usage of the key operator can all be operating parameters corresponding to the indicator field; the number of executions of the key operator, the duration occupied by the key operator each time it is executed, and the memory usage of the key operator can all be indicator values ​​corresponding to the indicator field.

[0226] In addition, the electronic device can determine the maximum memory usage of the node in the log information as the memory usage of the node; the duration of the node communicating with other nodes in the log information can be determined as the communication duration of the node; the processor utilization of the node in the log information can be determined as the processor utilization of the node. The maximum memory usage of the node, the duration of the node communicating with other nodes, and the processor utilization of the node can all be operating parameters corresponding to the indicator field; the memory usage of the node, the communication duration of the node, and the processor utilization of the node can all be indicator values ​​corresponding to the indicator field.

[0227] Case 2: Calculate the indicator value corresponding to the indicator field according to at least one operating parameter corresponding to the indicator field.

[0228] In this case, the number of operating parameters corresponding to the indicator field can be multiple, and the multiple operating parameters can be used to calculate the indicator value corresponding to the indicator field. The electronic device can calculate the indicator value corresponding to the indicator field through at least one operating parameter corresponding to the indicator field to achieve the purpose of determining the indicator value corresponding to the indicator field.

[0229] In this embodiment, the electronic device can obtain the duration of the node communicating with other nodes and the duration of the node both communicating with other nodes and performing calculations from the log information, and can calculate the ratio of the duration of the node both communicating with other nodes and performing calculations to the duration of the node communicating with other nodes, and can determine the ratio as the node's calculation communication ratio. The duration of the node communicating with other nodes and the duration of the node both communicating with other nodes and performing calculations can be the operating parameters corresponding to the indicator field; the node's calculation communication ratio can be the indicator value corresponding to the indicator field.

[0230] S805. Determine at least one indicator group from a plurality of indicator values ​​corresponding to the node set according to at least one indicator field to be monitored.

[0231] The indicator group includes the indicator value of each node in the node set corresponding to the same indicator field.

[0232] Specifically, the electronic device may determine the indicator values ​​corresponding to the same indicator field among the multiple indicator values ​​corresponding to the node set as an indicator group.

[0233] Exemplarily, it is assumed that the node set includes node 1, node 2, and node 3. The multiple index values ​​corresponding to the node set include the processor utilization of node 1, the communication duration of node 1, the calculation communication ratio of node 1, the processor utilization of node 2, the communication duration of node 2, the calculation communication ratio of node 2, the processor utilization of node 3, the communication duration of node 3, and the calculation communication ratio of node 3. Then the electronic device can determine the processor utilization of node 1, the processor utilization of node 2, and the processor utilization of node 3 as an index group; the communication duration of node 1, the communication duration of node 2, and the communication duration of node 3 as an index group; and the calculation communication ratio of node 1, the calculation communication ratio of node 2, and the calculation communication ratio of node 3 as an index group.

[0234] S806: For any indicator group, determine whether the indicator group includes an abnormal indicator value according to the indicator values ​​included in the indicator group.

[0235] If there is no indicator group including an abnormal indicator value, execute S807;

[0236] If there is an indicator group including an abnormal indicator value, execute S809.

[0237] For any indicator group, when the electronic device determines whether the indicator group includes an abnormal indicator value based on multiple indicator values ​​included in the indicator group, it can determine the mean and variance of the multiple indicator values ​​included in the indicator group; determine the alarm range corresponding to the indicator group based on the mean and variance; and determine whether the indicator group includes an abnormal indicator value based on the multiple indicator values ​​and the alarm range included in the indicator group.

[0238] The electronic device can calculate the average of multiple indicator values ​​in the indicator group by the following formula:

[0239]

[0240] in, is the mean of multiple indicator values ​​in the indicator group, n is the number of indicator values ​​in the indicator group, ∑X i It is the sum of the values ​​of each indicator in the indicator group.

[0241] The electronic device can calculate the variance of multiple indicator values ​​in the indicator group by the following formula:

[0242]

[0243] Among them, σ is the variance of multiple indicator values ​​in the indicator group, X i is the indicator value in the indicator group, is the mean of multiple indicator values ​​in the indicator group, and n is the number of indicator values ​​in the indicator group.

[0244] The alarm range may be the value range of the normal indicator value.

[0245] When the electronic device determines the alarm range, it can determine the product of the variance and the preset value; determine the difference between the mean and the product as the minimum value of the alarm range; and determine the sum of the mean and the product as the maximum value of the alarm range.

[0246] The preset value may be a pre-configured parameter value, and the specific size of the preset value may be configured according to actual needs, which is not limited in this embodiment. Optionally, the preset value may be 5.

[0247] Specifically, the minimum value of the alarm range can be: The maximum value of the alarm range can be: The warning range can be:

[0248] In this embodiment, the electronic device determines whether the indicator group includes an abnormal indicator value in at least the following two methods:

[0249] Method 1: Determine whether the indicator group includes abnormal indicator values ​​according to the alarm range corresponding to the indicator group and each indicator value.

[0250] In this case, the electronic device can determine whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group; if the target indicator value does not exist among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group does not include an abnormal indicator value; if the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group includes an abnormal indicator value, and the target indicator value is determined as the abnormal indicator value.

[0251] The target indicator value may be an indicator value outside the warning range, that is, the target indicator value may be an indicator value that is less than the minimum value of the warning range or greater than the maximum value of the warning range.

[0252] In this method, after the electronic device determines the alarm range, it can compare each indicator value in the indicator group with the alarm range respectively; if there is an indicator value that is the target indicator value, the electronic device can determine that the indicator group includes an abnormal indicator value.

[0253] In this method, the electronic device can directly determine the target index value as the abnormal index value, so that the abnormal index value can be determined more quickly.

[0254] Method 2: Determine whether the indicator group includes abnormal indicator values ​​based on the alarm range, each indicator value and historical indicator value corresponding to the indicator group.

[0255] In this method, the electronic device can determine whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group; if the target indicator value does not exist among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group does not include abnormal indicator values; if the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined whether the indicator group includes abnormal indicator values ​​based on the target indicator value.

[0256] When determining whether an indicator group includes an abnormal indicator value based on a target indicator value, the electronic device can determine a preset number of historical indicator values ​​corresponding to the target indicator value, and determine whether each historical indicator value is within a corresponding alarm range; if the preset number of historical indicator values ​​are all outside the corresponding alarm range, it is determined that the indicator group includes an abnormal indicator value, and the target indicator value is determined to be an abnormal indicator value; if there is a historical indicator value among the preset number of historical indicator values ​​that is within the corresponding alarm range, it is determined that the indicator group does not include an abnormal indicator value.

[0257] The preset number of historical indicator values ​​corresponding to the target indicator value may be the historical indicator values ​​of the node corresponding to the target indicator value during the preset number of historical training processes, wherein the historical indicator value and the target indicator value are indicator values ​​corresponding to the same indicator field.

[0258] The preset number of historical training processes may be a preset number of consecutive historical training processes adjacent to the current training process.

[0259] Exemplarily, assuming that the preset number is 3, the preset number of historical training processes can be 3 consecutive historical training processes adjacent to the current training process. The preset number of historical indicator values ​​corresponding to the target indicator value can include: the historical indicator value in the first historical training process before the current training process, the historical indicator value in the second historical training process before the current training process, and the historical indicator value in the third historical training process before the current training process.

[0260] Specifically, if a preset number of historical indicator values ​​corresponding to the target indicator value are all target indicator values ​​in the corresponding training process, the electronic device can determine that the target indicator value is an abnormal indicator value.

[0261] In this method, the electronic device can determine the abnormal index value according to the target index value and the historical index value, so that the determination of the abnormal index value is more accurate.

[0262] It should be noted that, during the actual implementation process, the method for determining the abnormal indicator value corresponding to each indicator group can be set according to actual needs.

[0263] Optionally, for the indicator group corresponding to the duration of each execution of the key operator, the indicator group corresponding to the memory usage of the node, the indicator group corresponding to the memory usage of the key operator in the node, the indicator group corresponding to the calculation communication ratio of the node, and the indicator group corresponding to the processor utilization of the node, method 1 can be used to determine whether there are abnormal indicator values ​​in the indicator group. For the indicator group corresponding to the communication duration of the node, method 2 can be used to determine whether there are abnormal indicator values ​​in the indicator group.

[0264] S807: Determine whether there is no abnormal index value among the multiple index values ​​corresponding to the node set.

[0265] In this embodiment, the method of S806 can be used to determine whether there is an abnormal index value in each index group corresponding to the node set. If there is no index group including an abnormal index value, the electronic device can determine that there is no abnormal index value in the multiple index values ​​corresponding to the node set.

[0266] S808: Determine whether there is no abnormal node in the node set.

[0267] In this embodiment, if there is no abnormal index value among the multiple index values ​​corresponding to the node set, the electronic device can determine that there is no abnormal node in the node set.

[0268] S809: Determine whether there is an abnormal index value among the multiple index values ​​corresponding to the node set.

[0269] In this embodiment, the method of S806 can be used to determine whether there are abnormal index values ​​in each index group corresponding to the node set. If any one or more index groups include abnormal index values, the electronic device can determine that there are abnormal index values ​​in the multiple index values ​​corresponding to the node set.

[0270] S810: Determine whether there is an abnormal node in the node set, and determine the node corresponding to the abnormal index value as an abnormal node.

[0271] In this embodiment, if there is an abnormal index value among the multiple index values ​​corresponding to the node set, the electronic device may determine that there is an abnormal node in the node set, and determine the node corresponding to the abnormal index value as an abnormal node.

[0272] S811. Issue an alarm.

[0273] It should be noted that the specific implementation method of S811 can refer to S504 and will not be repeated here.

[0274] In the abnormal node determination method provided in this embodiment, the electronic device can determine the program running on each node in the cluster; can determine one or more node sets in the cluster according to the program running on each node in the cluster; can obtain the log information of each node in the cluster; for any node set, can determine multiple index values ​​corresponding to the node set according to the log information of each node in the node set; can determine at least one index group from multiple index values ​​corresponding to the node set according to at least one index field to be monitored; for any index group, can determine whether the index group includes an abnormal index value according to the index value included in the index group; if there is no abnormal index value included in the index group, it can be determined that there is no abnormal index value in the multiple index values ​​corresponding to the node set, and there is no abnormal node in the node set; if there is an abnormal index value included in the index group, it can be determined that there is an abnormal index value in the multiple index values ​​corresponding to the node set, it can be determined that there is an abnormal node in the node set, and the node corresponding to the abnormal index value is determined as an abnormal node, and an alarm can be issued. In the above method, the electronic device can calculate and compare the multiple index values ​​in the index group to determine whether there is an abnormal index value in the index group, and then determine whether there is an abnormal node in the cluster. Through the above method, the electronic device can determine whether there is an abnormal node among the multiple nodes based on the log information of the multiple nodes running the same program code, thereby improving the efficiency of determining the abnormal nodes.

[0275] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 20 may include: a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can communicate; exemplarily, the processor 21 and the memory 22 communicate via a communication bus 23, the memory 22 is used to store computer execution instructions, and the processor 21 is used to call the computer execution instructions in the memory to execute the abnormal node determination method shown in any of the above method embodiments.

[0276] Optionally, the electronic device 20 may further include a communication interface, which may include a transmitter and / or a receiver.

[0277] The electronic device 20 may be the electronic device shown in any of the above method embodiments, and may execute the abnormal node determination method shown in any of the above method embodiments.

[0278] Optionally, the processor may be a CPU, a GPU, a baseboard management controller (BMC), other general-purpose processors, a digital signal processor (DSP), or an application-specific integrated circuit (ASIC). A general-purpose processor may be a microprocessor or the processor may be any conventional processor. The steps of the method disclosed in the embodiments of the present application may be directly implemented as a hardware processor, or may be implemented as a combination of hardware and software modules in the processor.

[0279] An embodiment of the present application provides a computer-readable storage medium, on which computer-executable instructions are stored; the computer-executable instructions are used to implement the abnormal node determination method as described in any of the above embodiments.

[0280] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above abnormal node determination method.

[0281] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned memory (storage medium) includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

[0282] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0283] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0284] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0285] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the embodiments of the present application are also intended to include these modifications and variations.

[0286] In the embodiments of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the element. The term "or" and its variations may refer to "and / or". In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. In the embodiments of the present application, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0287] After considering the invention disclosed in the specification and practice, those skilled in the art will easily think of other embodiments of the present application. The embodiments of the present application are intended to cover any variation, use or adaptive change of the embodiments of the present application, which follows the general principles of the embodiments of the present application and includes common knowledge or conventional technical means in the technical field that are not disclosed in the embodiments of the present application.

Claims

1. A method for determining abnormal nodes, characterized in that: include: Determine the programs running on each node in the cluster; Determine one or more node sets in the cluster according to the program running on each node in the cluster, wherein the program running on each node in the node set is the same; Obtaining log information of each node in the cluster; For any node set, whether there is an abnormal node in the node set is determined according to the log information of each node in the node set, and an alarm is issued when the abnormal node exists in the node set.

2. The method according to claim 1, characterized in that The step of determining, for any node set, whether there is an abnormal node in the node set according to the log information of each node in the node set includes: Determine a plurality of indicator values ​​corresponding to the node set according to the log information of each node in the node set; Determine whether there is an abnormal index value among the multiple index values ​​corresponding to the node set; If the abnormal index value does not exist in the multiple index values ​​corresponding to the node set, determining that the abnormal node does not exist in the node set; If the abnormal index value exists in the multiple index values ​​corresponding to the node set, it is determined that there is an abnormal node in the node set, and the node corresponding to the abnormal index value is determined as the abnormal node.

3. The method according to claim 2, characterized in that The determining, according to the log information of each node in the node set, a plurality of indicator values ​​corresponding to the node set comprises: For any node in the node set, determine at least one indicator value of the node according to the log information of the node; The multiple index values ​​corresponding to the node set include: the at least one index value of each node in the node set.

4. The method according to claim 3, characterized in that Determining at least one indicator value of the node according to the log information of the node includes: Determining at least one indicator field to be monitored; For any indicator field, obtaining at least one operating parameter corresponding to the indicator field from the log information of the node, and determining the indicator value corresponding to the indicator field according to the at least one operating parameter corresponding to the indicator field; The at least one indicator value of the node includes: an indicator value corresponding to the at least one indicator field determined according to the operating parameters of the node.

5. The method according to any one of claims 2 to 4, characterized in that: The determining whether there is an abnormal index value among the multiple index values ​​corresponding to the node set includes: According to at least one indicator field to be monitored, determining at least one indicator group from a plurality of indicator values ​​corresponding to the node set, the indicator group including indicator values ​​of each node in the node set corresponding to the same indicator field; For any indicator group, determining whether the indicator group includes an abnormal indicator value according to the indicator value included in the indicator group; If the abnormal index value is not included in the index group, determining that the abnormal index value does not exist in the multiple index values ​​corresponding to the node set; If there is an abnormal index value included in the index group, it is determined that the abnormal index value exists in the multiple index values ​​corresponding to the node set.

6. The method according to claim 5, characterized in that The step of determining, for any one indicator group, whether the indicator group includes an abnormal indicator value according to a plurality of indicator values ​​included in the indicator group, comprises: Determining the mean and variance of a plurality of indicator values ​​included in the indicator group; Determine the alarm range corresponding to the indicator group according to the mean and the variance; According to the multiple indicator values ​​included in the indicator group and the alarm range, it is determined whether the indicator group includes an abnormal indicator value.

7. The method according to claim 6, characterized in that The determining, according to the plurality of indicator values ​​included in the indicator group and the alarm range, whether the indicator group includes an abnormal indicator value comprises: Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group; If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value; If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined that the indicator group includes the abnormal indicator value, and the target indicator value is determined as the abnormal indicator value.

8. The method according to claim 6, characterized in that The determining, according to the plurality of indicator values ​​included in the indicator group and the alarm range, whether the indicator group includes an abnormal indicator value comprises: Determining whether there is a target indicator value outside the alarm range among the multiple indicator values ​​included in the indicator group; If the target indicator value does not exist in the plurality of indicator values ​​included in the indicator group, determining that the indicator group does not include the abnormal indicator value; If the target indicator value exists among the multiple indicator values ​​included in the indicator group, it is determined whether the indicator group includes an abnormal indicator value according to the target indicator value.

9. The method according to claim 6, characterized in that The determining, according to the target indicator value, whether the indicator group includes an abnormal indicator value comprises: Determine a preset number of historical indicator values ​​corresponding to the target indicator value, and determine whether each historical indicator value is within a corresponding alarm range; If the preset number of historical indicator values ​​are all outside the corresponding alarm range, determining that the indicator group includes the abnormal indicator value, and determining that the target indicator value is the abnormal indicator value; If there is a historical indicator value in the preset number of historical indicator values ​​that is within the corresponding alarm range, it is determined that the indicator group does not include the abnormal indicator value.

10. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

Images