Multi-dimensional fault sensing method and device for computing power computing node of intelligent computing center

Abstract

The invention provides a multi-dimensional fault sensing method and device for computing power computing nodes of an intelligent computing center, and belongs to the technical field of intelligent computing centers, intelligent computing centers and computing power infrastructure. The method comprises the steps that the operation state of at least one computing power computing node is monitored, and the computing power computing nodes with the abnormal operation state are sensed; taking the computing power computing node with the abnormal operation state as a target computing power computing node; obtaining target fault information of the target computing power computing node; the target fault information is compared with fault information in a preset database, the target fault dimension to which the fault of the target computing power computing node belongs is determined, the preset database comprises multiple pieces of fault information, and each piece of fault information corresponds to one fault dimension. The method can be used for solving the problem of how to realize multi-dimensional fault perception by an intelligent computing center.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent computing centers, intelligent computing centers, and computing power infrastructure, and particularly relates to a multi-dimensional fault perception method and device for computing power calculation nodes in an intelligent computing center. Background Art

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "intelligent computing centers" have emerged as the times require.

[0003] An "intelligent computing center" refers to a facility that uses large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, and mainly provides the required computing power, data, and algorithms for artificial intelligence applications (such as scenarios like artificial intelligence deep learning model development, model training, and model inference). An intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enabling.

[0004] The "intelligent computing center" includes, but is not limited to, the "intelligent computing center".

[0005] An "intelligent computing center", that is, an artificial intelligence computing center, is a type of computing power infrastructure that is based on artificial intelligence theory, adopts an artificial intelligence computing architecture, and provides computing power services, data services, and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing centers" and "intelligent computing centers", which is the ability of computer devices or computing / data centers to process information, the ability of computer hardware and software to cooperate to jointly execute a certain computing requirement, the computing ability to achieve the output of target results by processing information data, and a new type of productive force integrating information computing power, network carrying capacity, and data storage capacity, and mainly provides services to society through computing power infrastructure.

[0007] Currently, during the operation of an intelligent computing center, if a computing power calculation node fails, due to the lack of the ability to perceive faults in different dimensions, relevant personnel must first figure out which dimension the fault belongs to before they can perform targeted repairs, which will reduce the efficiency of fault repair and may lead to consequences such as long-term system downtime and data loss. Therefore, how to achieve multi-dimensional fault perception during the operation of an intelligent computing center is an urgent problem to be solved. Summary of the Invention

[0008] The present invention provides a multi-dimensional fault perception method and device for computing power calculation nodes in an intelligent computing center to solve the problem of how to achieve multi-dimensional fault perception in an intelligent computing center.

[0009] To solve the above technical problems, the present invention is implemented as follows:

[0010] In a first aspect, the present invention provides a multi-dimensional fault perception method for computing power calculation nodes in an intelligent computing center, including:

[0011] Step S1: Monitor the operating status of at least one computing power calculation node, sense the computing power calculation nodes with abnormal operating status, and use the computing power calculation nodes with abnormal operating status as target computing power calculation nodes;

[0012] Step S2: Obtain the target fault information of the target computing power calculation node;

[0013] Step S3: Compare the target fault information with the fault information in a preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs. The preset database includes multiple pieces of fault information, and each piece of fault information corresponds to a fault dimension.

[0014] Optionally, after the step S3, the method further includes:

[0015] Step S4: Trigger an alarm message for the target computing power calculation node. The alarm message includes a first message and a second message. The first message is used to indicate the target computing power calculation node, and the second message is used to indicate the target fault dimension.

[0016] Optionally, the fault dimension includes:

[0017] Network anomaly, disk failure, power outage event, GPU anomaly, memory anomaly, kernel anomaly, file system anomaly.

[0018] Optionally, when the target fault dimension is disk anomaly, the alarm message further includes a third message and a fourth message. The third message is used to indicate the disk where the anomaly occurs, and the fourth message is used to indicate the fault type of the disk. The fault type of the disk includes at least one of the following:

[0019] Hardware failure;

[0020] Software failure.

[0021] Optionally, when the target fault dimension is GPU anomaly, the alarm message further includes a fifth message and a sixth message. The fifth message is used to indicate the GPU where the anomaly occurs, and the sixth message is used to indicate the anomaly type of the GPU. The anomaly type of the GPU includes at least one of the following:

[0022] Link anomaly;

[0023] Power consumption anomaly;

[0024] GPU interruption.

[0025] Second aspect, the present invention provides a method for detecting the multi-dimensional fault perception ability of a computing power calculation node, including:

[0026] Step S5: Perform a target operation on the target computing power calculation node, where the target operation is used to cause a fault in the target computing power calculation node;

[0027] Step S6: Receive the detection information triggered by the target operation by the target computing power calculation node, where the detection information is used to indicate the fault dimension corresponding to the fault that occurs in the target computing power calculation node;

[0028] Step S7: Compare the fault dimension indicated by the detection information with the fault dimension corresponding to the target operation to generate comparison information, where the comparison information is used to indicate whether the fault dimensions are consistent or inconsistent.

[0029] Third aspect, the present invention provides a multi-dimensional fault perception device for a computing power calculation node of an intelligent computing center, including:

[0030] A monitoring module, configured to monitor the operating status of at least one computing power calculation node, sense the computing power calculation node with an abnormal operating status, and use the computing power calculation node with an abnormal operating status as the target computing power calculation node;

[0031] An acquisition module, configured to acquire the target fault information of the target computing power calculation node;

[0032] A determination module, configured to compare the target fault information with the fault information in a preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs, where the preset database includes a plurality of pieces of fault information, and each piece of fault information corresponds to a fault dimension.

[0033] Fourth aspect, the present invention provides an electronic device, including: a processor, a memory, and a program stored on the memory and executable on the processor, where when the program is executed by the processor, the steps of the method described in the first aspect above are implemented.

[0034] Fifth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0035] Sixth aspect, the present invention provides a computer program product, including computer instructions, where when the computer instructions are executed by a processor, the steps of the method described in the first aspect above are implemented.

[0036] In the present invention, by monitoring the operating status of at least one computing power calculation node, perceiving the computing power calculation node with abnormal operating status, and taking the computing power calculation node with abnormal operating status as the target computing power calculation node; obtaining the target fault information of the target computing power calculation node; comparing the target fault information with the fault information in the preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs. After determining that the target computing power calculation node is operating abnormally, comparing the target fault information with multiple pieces of fault information in the preset database can quickly determine the target fault dimension of the target fault information. In the subsequent process of fault repair, relevant personnel can directly repair the target computing power calculation node according to the target fault dimension, without having to determine the fault dimension of the target computing power calculation node through additional operations, which can improve the efficiency of fault repair and reduce the losses caused by the faults of the computing power calculation nodes during the operation of the intelligent computing center. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0038] Figure 1 is a flowchart of a multi-dimensional fault perception method for a computing power calculation node of an intelligent computing center provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of alarm information provided by an embodiment of the present invention;

[0040] Figure 3 is a flowchart of a detection method for the multi-dimensional fault perception ability of a computing power calculation node provided by an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of the structure of a multi-dimensional fault perception device for a computing power calculation node of an intelligent computing center provided by an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] First, the technical terms related to the present invention will be briefly described below.

[0045] The "computing power" referred to in the present invention means: the ability of a computer device or a computing / data center to process information, the ability of computer hardware and software to cooperate to jointly execute a certain computing requirement, the computing ability to achieve the output of a target result through processing information data, a new type of productive force integrating information computing power, network carrying capacity, and data storage capacity, and mainly providing services to society through computing power infrastructure.

[0046] The "computational power" (Computational Power, CP) referred to in the present invention means: the ability of a data center server to process data and achieve result output, a comprehensive index for measuring the computing ability of a data center, including general computing ability, supercomputing ability, and intelligent computing ability. The commonly used measurement unit is the number of floating-point operations per second (FLOPS, 1EFLOPS = 10^18 FLOPS), and the larger the value, the stronger the comprehensive computing ability. It is estimated that 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream laptops. The calculation formula is: CP = CP_general + CP_intelligent + CP_super.

[0047] The "carrying capacity" (Network Power, NP) referred to in the present invention means: the performance of the data transmission ability of computing power facilities, a comprehensive ability including network architecture, network bandwidth, transmission delay, intelligent management and scheduling, etc., involving network transmission inside and between data centers, and a comprehensive index for measuring network transmission scheduling ability.

[0048] The "storage power" (Storage Power, SP) referred to in the present invention means: the comprehensive ability of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon, a comprehensive index for measuring the data storage ability of a data center, including external storage devices such as storage arrays and server internal storage devices. The commonly used measurement unit for storage capacity is exabyte (EB, 1EB = 2^60 bytes), the commonly used measurement unit for performance is the number of read and write operations per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB), and the disaster recovery ratio is an important manifestation of security and reliability.

[0049] The "computing power infrastructure" referred to in the present invention means: a new type of information infrastructure integrating information computing power, network carrying capacity, and data storage capacity, which can realize the centralized computing, storage, transmission, and application of information.

[0050] The "new information infrastructure" described in the present invention refers to: mainly including network infrastructures such as 5G networks, fiber broadband networks, backbone networks, international communication networks, satellite Internet, etc., computing power infrastructures such as data centers, general computing power centers, intelligent computing centers, supercomputing centers, etc., and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0051] The "computing power" described in the present invention includes: general computing power, intelligent computing power, and super computing power.

[0052] The "general computing power" described in the present invention refers to: the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0053] The "intelligent computing power" described in the present invention refers to: for various artificial intelligence innovation applications, a computing platform deployed on a large scale based on special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit), such as natural language processing, machine vision, and so on.

[0054] The "super computing power" described in the present invention refers to: mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and processes extremely complex or data-intensive problems through a dedicated operating system. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, gene analysis, etc.

[0055] The "intelligent computing center" described in the present invention refers to: a facility that provides the required computing power, data, and algorithms mainly for artificial intelligence applications (such as scenarios like artificial intelligence deep learning model development, model training, and model inference) by using large-scale heterogeneous computing power resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enabling.

[0056] The "intelligent computing center" described in the present invention includes, but is not limited to, the "intelligent computing center".

[0057] The "intelligent computing center" described in the present invention, namely the artificial intelligence computing center, is a type of computing power infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications based on artificial intelligence theory and using an artificial intelligence computing architecture.

[0058] The "computing power center" described in the present invention refers to a facility mainly composed of infrastructure such as wind, fire, water, and electricity and IT software and hardware devices, which has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0059] The "supercomputing center" described in the present invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters and can provide functions such as large-scale computing, storage, and network services, and is widely used in application scenarios such as aerospace, national defense, oil exploration, climate modeling, and genome sequencing.

[0060] The "computing power resources" described in the present invention refer to technologies and facilities required for the development of the digital society, which have information computing, transmission, storage, and application capabilities, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and support and guarantee resources such as wind, fire, water, and electricity.

[0061] The "multi-dimension" described in the present invention means that in fault analysis, after classifying faults from different angles, different attributes, or different levels, different types of faults can be obtained, and each type corresponds to a dimension.

[0062] In the present invention, by monitoring the operating status of at least one computing power calculation node, perceiving the computing power calculation node with abnormal operating status, and taking the computing power calculation node with abnormal operating status as the target computing power calculation node; obtaining the target fault information of the target computing power calculation node; comparing the target fault information with the fault information in the preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs. After determining that the target computing power calculation node is operating abnormally, comparing the target fault information with multiple pieces of fault information in the preset database can quickly determine the target fault dimension of the target fault information. In the subsequent process of fault repair, relevant personnel can directly repair the target computing power calculation node according to the target fault dimension without having to determine the fault dimension of the target computing power calculation node through additional operations, which can improve the efficiency of fault repair and reduce the losses caused by the faults of the computing power calculation nodes.

[0063] Figure 1 It is a schematic flowchart of a multi-dimensional fault perception method for a computing power calculation node of an intelligent computing center provided by an embodiment of the present invention. As Figure 1 shown, the method includes the following steps:

[0064] Step S1: Monitor the operating status of at least one computing power calculation node, perceive the computing power calculation node with abnormal operating status, and take the computing power calculation node with abnormal operating status as the target computing power calculation node.

[0065] In this step, by monitoring the usage status of at least one computing power node in real time, the usage of computing power resources can be understood in real time, so that the abnormal usage status of the computing power node can be sensed in time, helping the operation and maintenance personnel quickly understand the fault situation and take corresponding measures, reducing the downtime and losses, and being able to respond to user needs and problems in time, improving the overall user experience.

[0066] Step S2: Obtain the target fault information of the target computing power node.

[0067] In this step, the monitoring of the computing power node can obtain the fault information of the target computing power node by directly retrieving the SNMP of the hardware system and the exporter of the computing node according to the Simple Network Management Protocol (SNMP) and the "Get Request" operation.

[0068] Step S3: Compare the target fault information with the fault information in the preset database to determine the target fault dimension to which the fault of the target computing power node belongs, where the preset database includes a plurality of fault information, and each fault information corresponds to a fault dimension.

[0069] In the present invention, according to the fault situations that occur during the historical operation of the computing power nodes in the intelligent computing center, the corresponding fault dimensions and fault information are summarized, and a preset database is established.

[0070] Optionally, the fault dimensions include: network anomaly, disk failure, power outage event, GPU anomaly, memory anomaly, kernel anomaly, file system anomaly.

[0071] After obtaining the target fault information, compare the target fault information with the fault information in the preset database one by one, find the fault information that is the same as the target fault information in the preset database, and determine the fault dimension corresponding to the fault information that is the same as the target fault information as the target fault dimension. When determining the target fault dimension of the target fault information, during the subsequent fault repair process, relevant personnel can directly repair the target computing power node according to the target fault dimension, without the need to determine the fault dimension of the target computing power node through additional operations, which can improve the efficiency of fault repair and reduce the losses caused by the fault of the computing power node.

[0072] Optionally, after the step S3, the method further includes:

[0073] Step S4: Trigger an alarm message for the target computing power node, where the alarm message includes a first piece of information and a second piece of information. The first piece of information is used to indicate the target computing power node, and the second piece of information is used to indicate the target fault dimension.

[0074] In this embodiment, the alarm message can be displayed in a preset display manner, where the display manner includes at least one of the following: displaying the alarm message using a visualization tool; receiving the alarm message in the form of an email, a text message, and / or a system notification.

[0075] Use a visualization tool to display the total number, severity (such as high, medium, low), and status (such as unprocessed, in progress, resolved) of alarms for a current computing power resource facility or all computing power resource facilities, and highlight the abnormal or faulty parts; and a chart can be formed to display the change in the number of alarms over a period of time to help identify peak periods and trends, and ensure that the visualization display can be updated in real time so that the operation and maintenance team can obtain the latest information in a timely manner; it is not limited to receiving the alarm message in the form of an email, a text message, and / or a system notification, and important alarm messages are promptly informed to a third-party terminal via email, text message, or in-app notification, such as Figure 2 shown, so as to more efficiently monitor and manage the system status, quickly identify and respond to potential problems, thereby improving the overall operation and maintenance efficiency and system reliability.

[0076] In this embodiment, when performing fault repair, relevant personnel can quickly determine the target computing power node from all computing power nodes in the intelligent computing center according to the received first piece of information, and then determine the fault dimension of the target computing power node according to the second piece of information. In the subsequent process of fault repair, relevant personnel can directly perform corresponding repairs on the target computing power node according to the target fault dimension without having to determine the fault dimension of the target computing power node through additional operations, which can improve the efficiency of fault repair.

[0077] Optionally, when the target fault dimension is disk anomaly, the alarm message further includes a third piece of information and a fourth piece of information. The third piece of information is used to indicate the disk where the anomaly occurs, and the fourth piece of information is used to indicate the fault type of the disk. The fault type of the disk includes at least one of the following:

[0078] Hardware failure;

[0079] Software failure.

[0080] Optionally, when the target fault dimension is a GPU anomaly, the alarm information further includes fifth information and sixth information. The fifth information is used to indicate the GPU where the anomaly occurs, and the sixth information is used to indicate the type of GPU anomaly. The type of GPU anomaly includes at least one of the following:

[0081] Link anomaly;

[0082] Power consumption anomaly;

[0083] GPU interruption.

[0084] In the present invention, when a fault occurs in a computing power calculation node in dimensions such as disk anomaly and GPU anomaly, the alarm information can further indicate the specific fault type in the corresponding dimension in addition to indicating the target computing power calculation node, and which device on the target computing power calculation node the fault occurs in, which can further improve the efficiency of fault repair.

[0085] The present invention also provides a method for detecting the multi-dimensional fault perception ability of a computing power calculation node. After applying the method of the embodiment shown in the present invention in an intelligent computing center, the computing power calculation node of the intelligent computing center has the multi-dimensional fault perception ability. For this ability, it can be verified whether it meets the standard through the method for detecting the multi-dimensional fault perception ability of the computing power calculation node shown in the embodiment, including the following steps: Figure 1 After applying the method of the embodiment shown in the present invention in an intelligent computing center, the computing power calculation node of the intelligent computing center has the multi-dimensional fault perception ability. For this ability, it can be verified whether it meets the standard through the method for detecting the multi-dimensional fault perception ability of the computing power calculation node shown in the embodiment, including the following steps: Figure 3 After applying the method of the embodiment shown in the present invention in an intelligent computing center, the computing power calculation node of the intelligent computing center has the multi-dimensional fault perception ability. For this ability, it can be verified whether it meets the standard through the method for detecting the multi-dimensional fault perception ability of the computing power calculation node shown in the embodiment, including the following steps:

[0086] Step S5: Perform a target operation on the target computing power calculation node, and the target operation is used to cause the target computing power calculation node to have a fault.

[0087] Step S6: Obtain the target fault dimension determined based on the multi-dimensional fault perception method of the computing power calculation node of the intelligent computing center as described in the embodiment; Figure 1 Step S6: Obtain the target fault dimension determined based on the multi-dimensional fault perception method of the computing power calculation node of the intelligent computing center as described in the embodiment;

[0088] Step S7: Compare the target fault dimension with the fault dimension corresponding to the target operation to generate comparison information, and the comparison information is used to indicate that the fault dimensions are the same or different.

[0089] The following example is used to illustrate step S5:

[0090] Example 1: Randomly select a computing power calculation node in the intelligent computing center and unplug the network cable.

[0091] Example 2: Randomly select a computing power calculation node in the intelligent computing center and replace a healthy disk with a faulty disk.

[0092] Example 3: Randomly select a computing power calculation node in the intelligent computing center and perform a power-off operation.

[0093] Example 4: Randomly select several GPU computing nodes in the intelligent computing center and repeatedly execute the GPU test program until a hardware exception occurs in the GPU.

[0094] Example 5: Randomly select several GPU computing nodes in the intelligent computing center, execute the GPU test program, and trigger an Out Of Memory (OOM) exception.

[0095] Example 6: Randomly select several GPU computing nodes in the intelligent computing center and disconnect the GPU interconnection link.

[0096] Example 7: Randomly select computing power nodes in the intelligent computing center and continuously write a large amount of data to the memory.

[0097] Example 8: Randomly select computing power nodes in the intelligent computing center and continuously write a large amount of data to the hard disk.

[0098] Example 9: Run a network test program in the intelligent computing center to cause network congestion.

[0099] Example 10: Randomly select computing power nodes in the intelligent computing center and run a test program that can trigger a kernel exception.

[0100] Example 11: Randomly select computing power nodes in the intelligent computing center and run a test program that can damage the file system.

[0101] Through different target operations, the target computing power nodes can be made to enter fault states in different dimensions. After that, obtain the target fault dimension determined by the method of the embodiment shown in Figure 1 Compare it with the fault dimension corresponding to the target operation to obtain comparison information. If the comparison information indicates that the fault dimensions are consistent, it means that the multi-dimensional fault perception ability of the computing power nodes meets the standard and can complete multi-dimensional fault perception; if the fault dimensions are inconsistent, it means that the multi-dimensional fault perception ability does not meet the standard yet.

[0102] Optionally, step S6 further includes:

[0103] Step S61: Obtain the alarm information determined by the multi-dimensional fault perception method of the computing power nodes based on the intelligent computing center. The alarm information includes the first information and the second information. Among them, the first information is used to indicate the target computing power node, and the second information is used to indicate the target fault dimension.

[0104] For the above 11 examples, the alarm information includes:

[0105] Example 1, the first information, the second information indicating the fault dimension is network exception, and the information indicating the fault type is network interruption.

[0106] Example 2, the first piece of information, the second piece of information indicating that the fault dimension is disk anomaly, and the information indicating that the fault type is disk hardware fault.

[0107] Example 3, the first piece of information, and the second piece of information indicating that the fault dimension is power outage event.

[0108] Example 4, the first piece of information, the second piece of information indicating that the fault dimension is GPU anomaly, the information indicating the GPU where the anomaly occurs, and the information indicating that the fault type of the GPU is low utilization or abnormal power consumption.

[0109] Example 5, the first piece of information, the second piece of information indicating that the fault dimension is GPU anomaly, the information indicating the GPU where the anomaly occurs, and the information indicating that the fault type of the GPU is OOM anomaly.

[0110] Example 6, the first piece of information, the second piece of information indicating that the fault dimension is GPU anomaly, the information indicating the specific link of the GPU where the anomaly occurs, and the information indicating that the fault type of the GPU is link anomaly.

[0111] Example 7, the first piece of information, and the second piece of information indicating that the fault dimension is memory anomaly.

[0112] Example 8, the first piece of information, the second piece of information indicating that the fault dimension is disk anomaly, and the information indicating that the fault type is disk software anomaly.

[0113] Example 9, the first piece of information, the second piece of information indicating that the fault dimension is network anomaly, and the information indicating that the fault type is network performance degradation or congestion.

[0114] Example 10, the first piece of information, and the second piece of information indicating that the fault dimension is kernel anomaly.

[0115] Example 11, the first piece of information, and the second piece of information indicating that the fault dimension is file system anomaly.

[0116] Through Figure 3 The method shown in the embodiments can verify whether the multi-dimensional fault perception ability of the intelligent computing center is qualified, thereby avoiding the waste of resources caused by the operation of the intelligent computing center with unqualified ability.

[0117] As Figure 4 shown, the embodiments of the present application further provide a multi-dimensional fault perception device 400 for a computing power calculation node of an intelligent computing center. The device 400 includes:

[0118] A monitoring module 401, configured to monitor the operating status of at least one computing power calculation node, sense the computing power calculation nodes with abnormal operating status, and use the computing power calculation nodes with abnormal operating status as target computing power calculation nodes;

[0119] An acquisition module 402, configured to acquire target fault information of the target computing power calculation node;

[0120] A determination module 403, configured to compare the target fault information with the fault information in a preset database to determine a target fault dimension to which the fault of the target computing power calculation node belongs, where the preset database includes a plurality of pieces of fault information, and each piece of fault information corresponds to a fault dimension.

[0121] Optionally, the apparatus 400 further includes:

[0122] An alarm module, configured to trigger an alarm message for the target computing power calculation node, where the alarm message includes a first message and a second message, the first message is used to indicate the target computing power calculation node, and the second message is used to indicate the target fault dimension.

[0123] Optionally, the fault dimension includes:

[0124] Network anomaly, disk failure, power outage event, GPU anomaly, memory anomaly, kernel anomaly, file system anomaly.

[0125] Optionally, when the target fault dimension is a disk anomaly, the alarm message further includes a third message and a fourth message, the third message is used to indicate the disk where the anomaly occurs, and the fourth message is used to indicate the fault type of the disk, and the fault type of the disk includes at least one of the following:

[0126] Hardware failure;

[0127] Software failure.

[0128] Optionally, when the target fault dimension is a GPU anomaly, the alarm message further includes a fifth message and a sixth message, the fifth message is used to indicate the GPU where the anomaly occurs, and the sixth message is used to indicate the anomaly type of the GPU, and the anomaly type of the GPU includes at least one of the following:

[0129] Link anomaly;

[0130] Power consumption anomaly;

[0131] GPU interruption.

[0132] The multi-dimensional fault perception apparatus 400 for the computing power calculation node of the intelligent computing center according to the embodiment of the present invention can implement each process of the multi-dimensional fault perception method for the computing power calculation node of the intelligent computing center according to the embodiment of the present invention, and achieve the same technical effect. To avoid repetition, it will not be described in detail here.

[0133] Please refer to Figure 5, the present invention also provides an electronic device, including a processor 502, a memory 501, and a computer program stored on the memory 501 and executable on the processor 502. When the computer program is executed by the processor 502, it implements each process of the multi-dimensional fault perception method of the computing power calculation node of the intelligent computing center in the above embodiments, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0134] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the multi-dimensional fault perception method of the computing power calculation node of the intelligent computing center in the above embodiments, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0135] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement each process of the Figure 1 multi-dimensional fault perception method of the computing power calculation node of the intelligent computing center shown in the above embodiments, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0136] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including that element.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit of the present invention and the scope protected by the claims, and all of them fall within the protection scope of the present invention.

Claims

1. A multi-dimensional fault perception method for computing nodes in an intelligent computing center, characterized in that: include: Step S1: monitor the operating status of at least one computing power computing node, sense the computing power computing node with abnormal operating status, and use the computing power computing node with abnormal operating status as the target computing power computing node; Step S2: Obtain target fault information of the target computing power calculation node; Step S3: Compare the target fault information with the fault information in a preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs, wherein the preset database includes multiple fault information, and each fault information corresponds to a fault dimension.

2. The method according to claim 1, characterized in that After step S3, the method further includes: Step S4: triggering alarm information for the target computing power calculation node, wherein the alarm information includes first information and second information, wherein the first information is used to indicate the target computing power calculation node, and the second information is used to indicate the target fault dimension.

3. The method according to claim 2, characterized in that The fault dimensions include: Network anomalies, disk failures, power outages, GPU anomalies, memory anomalies, kernel anomalies, and file system anomalies.

4. The method according to claim 3, characterized in that When the target fault dimension is a disk abnormality, the alarm information further includes third information and fourth information, the third information is used to indicate the disk where the abnormality occurs, and the fourth information is used to indicate the fault type of the disk, and the fault type of the disk includes at least one of the following: Hardware failure; Software failure.

5. The method according to claim 3, characterized in that: When the target fault dimension is a GPU abnormality, the alarm information further includes fifth information and sixth information, the fifth information is used to indicate the GPU in which the abnormality occurs, and the sixth information is used to indicate the abnormality type of the GPU, and the abnormality type of the GPU includes at least one of the following: Link abnormality; Abnormal power consumption; GPU interrupt.

6. A method for detecting multi-dimensional fault perception capability of computing nodes, characterized in that: The method comprises: Step S5: performing a target operation on the target computing power calculation node, wherein the target operation is used to cause the target computing power calculation node to fail; Step S6: Obtain a target fault dimension determined by the multi-dimensional fault perception method for computing nodes of a computing power center according to any one of claims 1 to 5; Step S7: Compare the target fault dimension with the fault dimension corresponding to the target operation to generate comparison information, where the comparison information is used to indicate whether the fault dimensions are consistent or inconsistent.

7. A multi-dimensional fault perception device for computing nodes in an intelligent computing center, characterized in that: include: A monitoring module, used to monitor the operating status of at least one computing power computing node, and sense the computing power computing node with abnormal operating status, and use the computing power computing node with abnormal operating status as a target computing power computing node; An acquisition module, used to acquire target fault information of the target computing power computing node; A determination module is used to compare the target fault information with the fault information in a preset database to determine the target fault dimension to which the fault of the target computing power calculation node belongs, wherein the preset database includes multiple fault information, and each fault information corresponds to a fault dimension.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the multi-dimensional fault perception method for computing power nodes of an intelligent computing center as described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-dimensional fault perception method for computing power nodes in an intelligent computing center according to any one of claims 1 to 5.

10. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the multi-dimensional fault perception method for computing power nodes of an intelligent computing center as described in any one of claims 1 to 5.

Images