Link fault positioning method and related equipment

Abstract

The link fault positioning method comprises the steps that a fault log of a hardware system in a to-be-detected computing system is obtained, the fault log records historical data and abnormal events of fault monitoring parameters of the hardware system and replacement records of a field replaceable unit (FRU) in the hardware system, and current data of the fault monitoring parameters of the hardware system are obtained, and then extracting a fault feature sequence of the hardware system according to the fault log and the current data of the fault monitoring parameters of the hardware system, and predicting the fault position of the FRU granularity according to the fault feature sequence of the hardware system. According to the method, the fault feature sequence is extracted from the fault log when the fault occurs and the current data of the fault monitoring parameters, the FRU corresponding to the fault feature sequence of the current fault is deduced, FRU-level link fault positioning is realized, and the positioning accuracy is improved. Moreover, the method is not limited to PCIe link fault positioning, universal link fault positioning can be realized, and the coverage rate of link fault positioning is improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on June 16, 2023, with application number 202310729844.9 and invention name “A Link Fault Location Method”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a link fault location method, system, baseboard management controller, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0003] With the continuous development of computer technology, diversified computing systems have been generated, such as various types of terminals, servers or computing device clusters. Computing systems usually include multiple types of hardware paths, including but not limited to the path from memory stick to central processing unit (CPU), the path from peripheral component interconnect express (PCIe) peripherals to CPU, the path from integrated circuit bus (I2C) peripherals to CPU, the path from serial attached SCSI interface (SAS) SAS / Serial ATA (Serial Advanced Technology Attachment) hard disk to CPU, the path between different CPUs of the same computing device, and the path between CPUs of different computing devices. The above hardware paths can be collectively referred to as links, and the failure occurring in the above links can be referred to as link failure.

[0004] The hardware scale of computing systems is getting larger and larger, the cables are getting more and more complex, and the link failure rate is getting higher and higher. When a link failure occurs, if the source of the failure cannot be quickly found, and active maintenance and timely repairs cannot be performed, it will be difficult for related services to recover quickly, which can cause business interruption or even paralysis, and even further spread of the failure, causing the entire system to crash. How to efficiently and accurately locate the fault when a link failure occurs in a computing system has become a key issue for maintenance personnel. Summary of the invention

[0005] The present application provides a link fault location method, which uses a link fault location model or a link fault location rule base constructed by a fault feature sequence marked with a faulty FRU to infer the FRU corresponding to the fault feature sequence of the current fault, thereby realizing FRU-level link fault location and improving the location accuracy. Moreover, the method is not limited to the location of a specific type of link fault, and can realize general link fault location, thereby improving the coverage of link fault location. The present application also provides a system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0006] In the first aspect, the present application provides a link fault location method. The method can be performed by a link fault location system. The link fault location system can be a software system, and the software system can adopt an online deployment (also called online deployment) mode, or an offline deployment (also called offline deployment) mode. When the software system adopts an online deployment mode, the software system can be deployed in a computing system, for example, a baseboard management controller (also called an out-of-band management system) deployed in a computing system. When the software system adopts an offline deployment mode, the software system can be deployed in a third-party computing device cluster with offline tools, such as a public cloud, an edge computing device, and an end-side computing device. The link fault location system can also be a hardware system, such as a baseboard management controller or a computing device cluster in a computing system. When hardware systems such as a baseboard management controller and a computing device cluster are running, the link fault location method of the present application can be executed.

[0007] Specifically, the link fault location system can obtain the fault log of the hardware system in the computing system to be detected, which records the historical data of the fault monitoring parameters of the hardware system, abnormal events and the replacement records of the field replaceable units FRU in the hardware system, and obtains the current data of the fault monitoring parameters of the hardware system. Then the link fault location system can extract the fault feature sequence of the hardware system based on the fault log and the current data of the fault monitoring parameters of the hardware system. Then the link fault location system can predict the fault location at the FRU granularity based on the fault feature sequence of the hardware system. Among them, the fault location at the FRU granularity is predicted in combination with the training results based on the labeled data, and the labeled data is labeled with the faulty FRU.

[0008] In some possible implementations, the training results include a link fault location model. Accordingly, the link fault location system can input the fault feature sequence of the hardware system into the link fault location model to obtain the fault probability of the FRU in the hardware system, and then determine the fault location of the FRU granularity according to the fault probability of the FRU in the hardware system.

[0009] Among them, the link fault location model can mine the hidden features of the fault feature sequence and map the hidden features to different FRU categories, so as to obtain the failure probability of FRU in the hardware system. The failure probability can more accurately predict the fault location of FRU granularity with high reliability.

[0010] In some possible implementations, when there is a FRU with a failure probability greater than a set threshold, the link fault location system can determine that the fault location includes the FRU with a failure probability greater than the set threshold. When the failure probability of the FRUs in the hardware system is not greater than the set threshold, the link fault location system can determine that the fault location is one or more of the first n FRUs in descending order of failure probability.

[0011] This method supports link faults that can accurately locate the source of the fault (for example, the probability of the fault is greater than the set threshold), and can directly output the fault location at the FRU granularity, thereby achieving accurate alarms; for link faults that are difficult to accurately locate the source of the fault, the Top N risk components can be recommended to meet the fault location requirements. Furthermore, this method also supports the generation of link fault analysis reports for operation and maintenance personnel to review.

[0012] In some possible implementations, a link fault location model is constructed in the following manner: obtaining training data, where the training data includes fault logs of hardware systems in a computing system deployed in a production environment, extracting fault feature sequences from the training data, and labeling faulty FRUs for the fault feature sequences to obtain labeled data, training a classifier based on the labeled data, and obtaining a link fault location model.

[0013] This method supports learning classification patterns from labeled data through supervised learning and other methods, and applying it to fault source classification can achieve fault location at the FRU granularity to meet the needs of refined fault location.

[0014] In some possible implementations, the training result includes a link fault location rule base. The link fault location rule base includes at least one link fault location rule. Accordingly, the link fault location system can match the fault feature sequence of the hardware system with at least one link fault location rule. When the fault feature sequence of the hardware system successfully matches the target rule in at least one link fault location rule, the link fault location system can determine the fault location at the FRU granularity from the target rule.

[0015] This method supports directly matching fault feature sequences with fault location rules in the fault location rule base, thereby achieving efficient, FRU-level fault location and meeting the requirements of accuracy and efficiency.

[0016] In some possible implementations, the training results include a link fault location model and a link fault location rule base. The link fault location rule base includes at least one link fault location rule. Accordingly, the link fault location system can match the fault feature sequence of the hardware system with at least one link fault location rule. When the match fails, the link fault location system inputs the fault feature sequence of the hardware system into the link fault location model to obtain the fault location at the FRU granularity.

[0017] The method supports the collaborative completion of FRU-granular link fault location based on link fault location rules and link fault location models, thereby achieving a balance between the accuracy and efficiency of fault location.

[0018] In some possible implementations, the link fault location system can also generate a link topology of the hardware system based on the networking information of the hardware system, and the link topology records the link type of the link in the hardware system and the FRUs included in the link of at least one link type. Accordingly, the link fault location system can predict the fault location at the FRU granularity based on the fault feature sequence of the hardware system and the link topology of the hardware system.

[0019] This method can reduce the matching range or search range of fault link location by adding prior information of link topology, thereby improving the efficiency and accuracy of link fault location.

[0020] In some possible implementations, fault monitoring parameters include one or more of temperature, current, voltage, or in-position state, plugged-in state, and abnormal events include one or more of sideband detection abnormality, lighting abnormality, reliability availability, serviceability (RAS) abnormality, and RAS abnormality. RAS abnormality includes data access abnormality during the operation of chips, controllers, buses, or input / output (IO) peripherals.

[0021] This method collects parameters such as temperature, current, voltage, or in-place status, plugged-in status, and events such as sideband detection anomalies, lighting anomalies, and RAS anomalies as positioning inputs, which are easy to obtain and do not increase the operation of maintenance personnel.

[0022] In some possible implementations, the link fault location system can extract one or more of the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event from the fault log, and construct a fault feature sequence of the hardware system based on the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event and the historical data and current data of the fault monitoring parameters.

[0023] This method constructs a structured fault feature sequence by extracting the number of occurrences, occurrence time, event subject or link where the event is located, which is conducive to subsequent model reasoning or rule matching and provides assistance for fault location at the FRU granularity.

[0024] In a second aspect, the present application provides a link fault location system. The system comprises:

[0025] A parameter and log collection module, used to obtain a fault log of a hardware system in a computing system to be detected, wherein the fault log records historical data of fault monitoring parameters of the hardware system, abnormal events, and replacement records of field replaceable units (FRUs) in the hardware system, and obtains current data of fault monitoring parameters of the hardware system;

[0026] A preprocessing module, used for extracting a fault feature sequence of the hardware system according to the fault log and current data of the fault monitoring parameters of the hardware system;

[0027] A link fault location module is used to predict the fault location of FRU granularity according to the fault feature sequence of the hardware system, wherein the fault location of FRU granularity is predicted in combination with the training result based on the labeled data, and the labeled data is labeled with the faulty FRU.

[0028] In some possible implementations, the training result includes a link fault location model;

[0029] The link fault locating module is specifically used for:

[0030] Inputting the fault feature sequence of the hardware system into the link fault location model to obtain the failure probability of the FRU in the hardware system;

[0031] According to the failure probability of the FRU in the hardware system, the failure location of the FRU granularity is determined.

[0032] In some possible implementations, the link fault locating module is specifically used to:

[0033] When there is a FRU with a failure probability greater than a set threshold, it is determined that the fault location includes the FRU whose failure probability is greater than the set threshold; when the failure probability of all FRUs in the hardware system is not greater than the set threshold, it is determined that the fault location is one or more of the first n FRUs sorted from high to low in failure probability.

[0034] In some possible implementations, the link fault location system further includes:

[0035] A construction module is used to obtain training data, wherein the training data includes fault logs of hardware systems in a computing system deployed in a production environment, extract fault feature sequences from the training data, and mark faulty FRUs according to the fault feature sequences to obtain the marked data, train a classifier based on the marked data, and obtain the link fault location model.

[0036] In some possible implementations, the training result includes a link fault location rule base, and the link fault location rule base includes at least one link fault location rule;

[0037] The link fault locating module is specifically used for:

[0038] Matching the fault feature sequence of the hardware system with the at least one link fault locating rule;

[0039] When the fault feature sequence of the hardware system successfully matches the target rule in the at least one link fault location rule, the fault location at the FRU granularity is determined from the target rule.

[0040] In some possible implementations, the training result includes a link fault location model and a link fault location rule base, and the link fault location rule base includes at least one link fault location rule;

[0041] The link fault locating module is specifically used for:

[0042] Matching the fault feature sequence of the hardware system with the at least one link fault locating rule;

[0043] When the matching fails, the fault feature sequence of the hardware system is input into the link fault location model to obtain the fault location at the FRU granularity.

[0044] In some possible implementations, the preprocessing module is further used to:

[0045] Generate a link topology of the hardware system according to the networking information of the hardware system, wherein the link topology records link types of links in the hardware system and FRUs included in links of at least one link type;

[0046] The link fault locating module is specifically used for:

[0047] The fault location at FRU granularity is predicted according to the fault feature sequence of the hardware system and the link topology of the hardware system.

[0048] In some possible implementations, the fault monitoring parameters include one or more of temperature, current, voltage, or in-position state and plugged-in state; the abnormal events include one or more of sideband detection abnormality, lighting abnormality, reliability availability, serviceability (RAS) abnormality; and the RAS abnormality includes data access abnormality during the operation of a chip, controller, bus, or input / output (IO) peripheral.

[0049] In some possible implementations, the preprocessing module is specifically used to:

[0050] Extract one or more of the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event from the fault log;

[0051] The fault feature sequence of the hardware system is constructed according to one or more of the number of occurrences, the occurrence time, the event subject or the link where the event is located, and the historical data and current data of the fault monitoring parameters of the abnormal event.

[0052] In a third aspect, the present application provides a baseboard management controller. The baseboard management controller includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the baseboard management controller performs the link fault location method described in the first aspect or any implementation of the first aspect.

[0053] In a fourth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the link fault location method as described in the first aspect or any implementation of the first aspect.

[0054] In a fifth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the link fault location method described in the first aspect or any implementation of the first aspect.

[0055] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the link fault location method described in the first aspect or any one of the implementations of the first aspect.

[0056] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the architecture of a link fault location system provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of the structure of a link fault location system provided in an embodiment of the present application;

[0059] Figure 3 A flowchart of a link fault locating method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a process for training a link fault location model provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of a link fault location method provided in an embodiment of the present application;

[0062] Figure 6 Link topology diagrams of different types of computing systems provided by embodiments of the present application;

[0063] Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0065] Fig. 9 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0066] Fig.10 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0068] First, some technical terms involved in the embodiments of the present application are introduced.

[0069] A link refers to the hardware path between components in a computing system (such as a terminal, server, or computing device cluster), such as the hardware path from the CPU to other components, or the hardware path from the baseboard management controller (BMC) to other components. Other components can be PCIe peripherals, I2C peripherals, SAS / SATA hard drives, or other CPUs. The hardware path is used to carry data and control signal flows between the CPU, BMC, and other components.

[0070] Links can be divided into high-speed links and low-speed links according to the signal transmission rate. Among them, high-speed links are links with higher signal transmission rates, usually used to carry data signal flow, such as high-speed links including memory links (such as links from CPU to memory), PCIe links (such as links from CPU to PCIe peripherals), SAS / SATA links (such as links from CPU to SAS / SATA hard disks), compute express links (Compute Express Link, CXL) links, Unified Bus (Unified BUS, UB) links, and Hisilicon Coherent Connect Specification (HCCS) links. Low-speed links are links with lower signal transmission rates, usually used to carry control signal flow, such as I2C, Low PinCount Bus (LPC), and Serial Peripheral Interface (SPI) links.

[0071] When a link failure occurs in a computing system, if the source of the failure is not quickly found for proactive maintenance and timely repair, related services cannot be quickly restored, which may cause business interruption or even paralysis, and may even further spread the failure and cause the computing system to crash. The industry provides a link fault location method. Specifically, when a computing system fails, the PCIe link layer anomaly or message anomaly (such as packet loss, message check anomaly, message timeout) can be identified based on the Advanced Error Reporting (AER) fault information reported by the PCIe peripheral (also called the endpoint), so as to determine whether it is a link failure, and make recommendations for locating the source of the fault based on the single point problem recorded by the endpoint.

[0072] However, the above-mentioned link fault location method can only infer whether it is a PCIe link fault based on whether the PCIe peripheral has an error. The judgment logic for the fault source is fixed, which leads to low accuracy in locating the fault source. In most scenarios, the fault source cannot be located, and can only be roughly output as a PCIe link fault. It is difficult to determine whether the fault source on the link belongs to the CPU controller, cable, backplane, adapter card or terminal device.

[0073] In view of this, the present application provides a link fault location method. The method can be performed by a link fault location system. The link fault location system can be a software system, and the software system can adopt an online deployment (also called online deployment) mode, or an offline deployment (also called offline deployment) mode. When the software system adopts an online deployment mode, the software system can be deployed in a computing system, for example, a BMC (also called an out-of-band management system) deployed in a computing system. When the software system adopts an offline deployment mode, the software system can be deployed in a third-party computing device cluster with offline tools, such as a public cloud, an edge computing device, and an end-side computing device. The link fault location system can also be a hardware system, such as a BMC in a computing system or a computing device cluster. When hardware systems such as BMC and computing device clusters are running, the link fault location method of the present application can be executed.

[0074] Specifically, the link fault location system can obtain a fault log of a hardware system in a computing system to be detected, which records historical data of fault monitoring parameters of the hardware system, abnormal events, and replacement records of field replaceable units (FRU) in the hardware system, as well as current data of fault monitoring parameters of the hardware system, and then extract a fault feature sequence of the hardware system based on the fault log and the current data of the fault monitoring parameters of the hardware system, and then predict the fault location at the FRU granularity based on the fault feature sequence of the hardware system. Among them, the fault location at the FRU granularity is predicted in combination with the training results based on the labeled data. The labeled data is labeled with a faulty FRU, for example, a fault feature sequence labeled with a faulty FRU. The training results based on the labeled data may include, but are not limited to, a link fault location model or a link fault location rule base.

[0075] The method extracts the fault feature sequence from the fault log when the fault occurs and the current data of the fault monitoring parameters, and then uses the training results based on the labeled data, such as the link fault location model or link fault location rule base built based on the fault feature sequence of the labeled faulty FRU, to infer the FRU corresponding to the fault feature sequence of the current fault, thereby realizing the link fault location at the FRU level and improving the location accuracy. Moreover, the method is not limited to the location of PCIe link faults, and can realize general link fault location, improving the coverage of link fault location.

[0076] In order to make the technical solution of the present application clearer and easier to understand, the architecture of the link fault location system of the present application is first introduced below with reference to the accompanying drawings.

[0077] See also Figure 1 The schematic diagram of the architecture of a link fault location system is shown, and the link fault location system 10 is used to locate link faults in a computing system. The computing system can be a multi-server or other integrated computing system such as a cross-architecture ordinary server, a storage server, a high performance computing (HPC) server, etc. with a main processor, a coprocessor or a BMC. The computing system can also be a terminal with an embedded controller, including but not limited to a smart phone and a tablet computer. The computing system can also be a data center or a computing device cluster formed by multiple computing devices.

[0078] The computing system includes a hardware system, firmware and an operating system. The hardware system includes a power module, a processor (such as a CPU), a baseboard management controller BMC, and in some examples, the hardware system may also include high / low speed peripherals. The power module is connected to the processor, BMC, and high / low speed peripherals respectively, and is used to power the processor, BMC, and high / low speed peripherals. The processor is connected to the BMC and high / low speed peripherals respectively, and the BMC is connected to the high / low speed peripherals.

[0079] Among them, BMC is a small dedicated processor. BMC can monitor the operating status of the computing system using sensors or by analyzing in-band logs. In-band logs include but are not limited to operating system logs, driver logs or firmware logs. In addition, BMC can communicate with system administrators through independent lines. BMC is usually contained in the motherboard or the main circuit board of the monitored computing device. BMC's sensors are used to measure internal physical variables, including but not limited to temperature, humidity, power supply voltage, and fan speed. When the above variables exceed the set range, BMC can notify the system administrator, who can make adjustments through remote control. It should be noted that BMC also supports local control of computing systems.

[0080] Firmware is a kind of fixed code. When running, firmware occupies non-volatile memory (NVM) and does not require additional memory and processor resources. It is estimated to be used to perform basic tests, initialize the hardware system, and then load the operating system loader from the bootable device (such as a hard drive) into the memory. When the processor of the hardware system includes a main processor and a coprocessor, the firmware may include the main processor firmware or the coprocessor firmware.

[0081] An operating system (OS) is a set of interrelated system software programs that manage and control computer operations, use and run hardware and software resources, and provide public services to organize user interactions. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. The operating system also provides an operating interface for users to interact with the system.

[0082] The BMC in the hardware system can also communicate with the firmware and operating system to obtain the fault logs reported by the firmware and operating system for link fault location. Figure 1 In the example, the link fault location system 10 is deployed in the BMC, and the BMC runs the program code of the link fault location system 10, thereby executing the link fault location method of the present application, realizing link fault location at FRU granularity, and improving the location accuracy.

[0083] The link fault location system 10 may include a parameter and log collection module and a link fault location module. The parameter and log collection module is used to obtain the fault log of the hardware system in the computing system to be detected, and to obtain the current data of the fault monitoring parameters of the hardware system. Among them, the fault log records the historical data of the fault monitoring parameters of the hardware system, abnormal events and the replacement records of FRUs in the hardware system. The link fault location module is used to predict the fault location of FRU granularity based on the fault feature sequence of the hardware system extracted from the fault log and the current data of the fault monitoring parameters of the hardware system. Among them, the fault location of FRU granularity can be predicted in combination with the training results based on the labeled data (such as at least one of the link fault location model or the link fault location rule base).

[0084] It should be noted that fault logs can be collected and reported by hardware systems, firmware, and operating systems. Based on this, parameter and log collection modules can also be deployed in hardware systems, firmware, and operating systems.

[0085] Figure 1 The link fault location system 10 is deployed in the BMC for illustration only. In other possible implementations of the embodiment of the present application, the link fault location system 10 may also be an offline link fault location tool.

[0086] The following takes the deployment of the link fault location system 10 in the BMC as an example to describe the structure of the link fault location system 10 in detail from the perspective of functional modularization.

[0087] See also Figure 2The schematic diagram of the structure of a link fault location system 10 is shown, and the link fault location system 10 includes a parameter and log collection module 102, a preprocessing module 104 and a link fault location module 106. Furthermore, the link fault location system 10 may also include an output module 108 and a construction module 109. The functions of each module are introduced below.

[0088] The parameter and log collection module 102 is used to obtain the fault log of the hardware system in the computing system to be detected, and to obtain the current data of the fault monitoring parameters of the hardware system. Among them, the fault log records the historical data of the fault monitoring parameters of the hardware system, abnormal events and the replacement records of FRUs in the hardware system. In this way, it is possible to provide as comprehensive operating environment information and abnormal data as possible for link fault location.

[0089] The fault log may be collected and reported by the hardware system, firmware or operating system (for example, the driver in the operating system). The firmware may include the Basic Input Output System (BIOS) firmware. The fault monitoring parameters of the hardware system may be the fault monitoring parameters of the high-speed link and the low-speed link of the hardware system. Among them, the high-speed link may include the link where the high-speed bus device is located. Figure 1 As shown, high-speed bus devices include but are not limited to CPU, Mem, PCIe peripherals, CXL peripherals, HCCS peripherals, SAS hard disks, SATA hard disks or USB peripherals. Low-speed bus devices include but are not limited to I2C peripherals, LPC peripherals, SPI peripherals, serial general purpose input / output (SGPIO) peripherals or universal serial bus (USB) peripherals.

[0090] In some possible implementations, the fault monitoring parameters may include temperature, current, voltage (such as power supply voltage) or one or more of in-position state and plug-in stable state, and the abnormal events may include one or more of sideband detection abnormality, lighting abnormality, reliability availability serviceability (RAS) abnormality. Among them, RAS abnormality includes data access abnormality during the operation of chip, controller, bus or IO peripheral, wherein the data access abnormality may be error detection and correction abnormality or illegal access abnormality. Among them, error detection and correction abnormality includes but is not limited to error checking and correcting (ECC), parity check parity, cyclic redundancy check (CRC). Among them, the IO peripheral may be a PCIe peripheral, and the PCIe peripheral is located in the terminal (endpoint, EP) device in the PCIe subsystem. Therefore, in the PCIe subsystem, the PCIe peripheral abnormality may be an EP abnormality. The PCIe subsystem may also include a root component (RP), and the root component abnormality is the RP abnormality. In some examples, the IO peripheral may also be an IO peripheral based on a protocol interface such as UB, CXL, SAS, SATA, USB, I2C, SPI, etc.

[0091] The preprocessing module 104 is used to extract the fault feature sequence of the hardware system based on the current data of the fault log and the fault monitoring parameters of the hardware system. In order to improve the efficiency and quality of feature extraction, the preprocessing module 104 can also first clean the current data of the fault log and the fault monitoring parameters. Among them, the cleaning of the current data of the fault log and the fault monitoring parameters may include classification, structuring, and filtering. Among them, filtering refers to filtering data that does not meet the requirements, such as incomplete data or erroneous data. Further, when the fault feature sequence is provided to the back end for use in the form of a data interface, the preprocessing module 104 is also used to abstract and standardize the data format of the fault feature sequence after extracting the fault feature sequence, so that the link fault can be located based on the abstracted or standardized fault feature sequence in the future. Among them, abstraction refers to separating data from implementation, presenting necessary information without presenting details, and standardization refers to unifying data with the same physical meaning represented by different fields.

[0092] The link fault location module 106 is used to predict the fault location at the FRU granularity according to the fault feature sequence of the hardware system. Among them, the fault location at the FRU granularity is predicted in combination with the training results based on the labeled data. Among them, the labeled data is labeled with a faulty FRU, for example, it can be a fault feature sequence labeled with a faulty FRU. The training result based on the labeled data can be a link fault location model or a link fault location rule base. In specific implementation, the link fault location module 106 performs precise matching of the fault source through the link fault location model or the link fault location rule base according to the fault feature sequence of the hardware system. When the match is successful, the fault source is determined to be a successfully matched FRU. When the match is unsuccessful, the fault source is determined to be one or more of the top n FRUs in the order of matching degree from high to low. The top n FRUs in the order of matching degree from high to low are also called risk components or suspected fault sources.

[0093] In some possible implementations, the preprocessing module 104 is also used to generate a link topology of the hardware system based on the networking information of the hardware system. The link topology records the link type of the link in the hardware system and the FRU included in the link of at least one link type. The preprocessing module 104 can obtain the networking information in real time, thereby generating a full amount of link topology. Correspondingly, the link fault location module 106 is specifically used to predict the fault location of the FRU granularity according to the fault feature sequence of the hardware system and the link topology of the hardware system. For example, the link fault location module 106 can determine the link fault location rule related to the link topology according to the link topology of the hardware system, match the fault feature sequence of the hardware system with the link fault location rule related to the link topology, and obtain the fault location of the FRU granularity. For another example, the link fault location module 106 can input the link topology of the hardware system into the link fault location model, and the link fault location model uses the link topology as a priori information for reasoning to obtain the fault location of the FRU granularity. In this way, the matching range or search range can be reduced, and the efficiency and accuracy of link fault location can be improved.

[0094] The output module 108 is used to output the link fault location result. Specifically, when the link fault location module 106 accurately matches successfully, it can be determined that the fault location result is a successfully matched FRU, and the output module 108 can directly output the successfully matched FRU. Furthermore, the output module 108 can also issue accurate alarms based on the successfully matched FRU. For example, the output module 108 can notify the system administrator of the above-mentioned successfully matched FRU fault, so that the system administrator can shield or replace the above-mentioned FRU to achieve fault recovery. When the link fault location module 106 fails to accurately match, the output module 108 can output n FRUs that are suspected to be the source of the fault. Regardless of whether the accurate match is successful, the output module 108 can generate a link fault analysis report based on the above-mentioned link fault location result, and output the link fault analysis report for reference during operation and maintenance.

[0095] The construction module 109 is used to construct a link fault location model or a link fault location rule base. Specifically, the construction module 109 can obtain training data, and the training data includes the fault log of the hardware system in the computing system deployed in the production environment (also called the log of the existing network problem case), and then extract the fault feature sequence from the training data, and mark the fault FRU according to the fault feature sequence, and train the classifier according to the fault feature sequence of the marked fault FRU to obtain the link fault location model. It should be noted that when the construction module 109 trains the classifier to obtain the link fault location model, it can use offline fault logs, and perform artificial intelligence (AI) training through offline training methods (such as offline machine learning) to obtain a link fault location model based on a machine learning algorithm. The construction module 109 can extract link fault location rules according to the fault feature sequence of the marked fault FRU, and generate a link fault location rule base. Among them, the construction module 109 can match the fault feature sequence of the marked fault FRU with the rule template to extract the link fault location rule. For example, for existing network problem cases with complete experience and direct link fault location (i.e., the faulty FRU can be determined), the construction module 109 can generate link fault location rules based on the fault feature sequence marked with the faulty FRU and combined with the rule template; for existing network problem cases without complete experience, the construction module 109 can process the fault feature sequence as a priori features, or add labels to enter the link fault location model for noise reduction, so as to be used for link fault location model training. Prior features refer to the prior knowledge of a sample or a class of samples in machine learning, which can help optimize the performance of the model during the training process. Generally speaking, prior features are derived from expert knowledge or historical data, which can help reduce the difficulty of model training and improve the generalization ability of the model.

[0096] It should be noted that when the link fault location system 10 is deployed offline, the structure of the link fault location system 10 can refer to Figure 2 , I will not go into details here.

[0097] Based on the aforementioned link fault location system 10, the present application further provides a link fault location method. The link fault location method of the present application is described in detail below in conjunction with an embodiment.

[0098] See also Figure 3 A flow chart of a link fault location method is shown, the method comprising the following steps:

[0099] S302 : The link fault locating system 10 obtains a fault log of a hardware system in the computing system to be detected.

[0100] A computing system can be a computing device such as a server or terminal, or a computing device cluster. Among them, a terminal can be a desktop, a laptop, or an embedded computing device such as a tablet or a smart phone. The hardware system in a computing system refers to a system formed by hardware devices, including processors, IO peripherals, BMC and other components. The hardware paths between components are called links of the computing system.

[0101] When a link failure occurs in a hardware system, the computing system may generate a fault log. For example, the hardware system, firmware, and operating system may generate a fault log. The link fault location system 10 obtains the fault log reported by the computing system, for example, the fault log reported by the hardware system, firmware, and operating system.

[0102] Among them, the fault log records the historical data of the fault monitoring parameters of the hardware system, abnormal events and the replacement records of the field replaceable units FRU in the hardware system. In some possible implementations, the fault monitoring parameters may include one or more of temperature, current, voltage, or in-place state and plugged-in state. Abnormal events may include one or more of sideband detection abnormalities, lighting abnormalities, and RAS abnormalities. Among them, RAS abnormalities include data access abnormalities during the operation of chips, controllers, buses or IO peripherals, wherein data access abnormalities may be error detection and correction abnormalities or illegal access abnormalities. Error detection and correction abnormalities include but are not limited to ECC, parity check parity, and cyclic redundancy check CRC.

[0103] S304: The link fault locating system 10 obtains current data of fault monitoring parameters of the hardware system.

[0104] Specifically, the link fault location system 10 can obtain the current data of the fault monitoring parameters of the hardware system in real time through sensors. For example, the link fault location system 10 can obtain the current value of the temperature of the hardware system in real time through a temperature sensor.

[0105] It should be noted that the above S302 and S304 can be executed in parallel, or can be executed successively in a set order. For example, the link fault locating method of the present application can also execute S304 first and then execute S302. The embodiment of the present application does not limit this.

[0106] S306 . The link fault locating system 10 extracts a fault feature sequence of the hardware system according to the fault log and current data of the fault monitoring parameters of the hardware system.

[0107] In specific implementation, the link fault location system 10 can perform feature recognition based on the current data of the fault log and the fault monitoring parameters of the hardware system to obtain features of different feature categories, and then obtain a fault feature sequence based on the features of different feature categories. In some examples, the feature category may include one of BMC alarms, equipment-specific indicators, startup / operation logs, fault diagnosis management (FDM) information, or maintenance logs.

[0108] Different feature categories can include the following fault features:

[0109] Table 1 Feature categories and corresponding feature examples

[0110] Feature Category Feature Examples BMC Alarm Alarm effectiveness, occurrence times, subject FRU type, subject number statistics, subject link association information Equipment-specific indicators SMART, GList, PHY CRC, ECC, Health status, Self Test Status Startup / Runtime Log PHY down, Init fail, Reset fail FDM Information Caterr, IERR, AER, ARER, UCE, CE exceed Maintenance Log Voltage exceed

[0111] Among them, SMART stands for "Self-Monitoring Analysis and Reporting Technology", which is an automatic hard disk status detection and early warning system and specification. Through the detection instructions in the hard disk hardware, the operation of the hard disk hardware such as the head, disk, motor, and circuit is monitored, recorded, and compared with the preset safety values ​​set by the manufacturer. If the monitoring situation is about to or has exceeded the safety range of the preset safety value, the host's monitoring hardware or software can automatically warn the user and perform minor automatic repairs to ensure the safety of the hard disk data in advance. GList stands for Grown-defect List, which is a detection tool made before the hard disk leaves the factory. The purpose is to record the defect information when the hard disk is manufactured. During use, if the hard disk has problems such as bad sectors, it can be marked in Glist to avoid accessing the bad sectors again in the subsequent reading and writing process, resulting in data loss or other problems. PHY CRC and PHY down in PHY means physical layer. Caterr and IERR are timeout errors. UCE stands for Uncorrectable Error and CE stands for Correctable Error, respectively. Exceed means exceeding the threshold. CE exceed means CE exceeding the threshold, and Voltage exceed means voltage exceeding the threshold.

[0112] The link fault location system 10 can extract one or more of the number of occurrences, occurrence time, event subject, or link where the event is located of the abnormal event from the fault log. For example, the link fault location system can extract the number of occurrences, occurrence time, event subject, or link where the abnormal event is located based on the features of different feature categories identified from the fault log, and then construct a fault feature sequence of the hardware system based on the number of occurrences, occurrence time, event subject, or link where the abnormal event is located and one or more of the historical data and current data of the fault monitoring parameters.

[0113] S308 . The link fault location system 10 predicts the fault location at the FRU granularity according to the fault feature sequence of the hardware system.

[0114] Among them, the fault location at the FRU granularity is predicted in combination with the training results based on the labeled data. The labeled data is labeled with a faulty FRU. For example, the labeled data can be a fault feature sequence labeled with a faulty FRU. Specifically, the link fault location system 10 can obtain training data, which can include the fault log of the hardware system in the computing system deployed in the production environment (also referred to as the existing network problem case), and then the link fault location system 10 can perform feature extraction on the above-mentioned existing network problem case to obtain a fault feature sequence, and then the link fault location system 10 can receive the labeling information of the above-mentioned fault feature sequence, thereby obtaining a fault feature sequence labeled with a faulty FRU. The training result based on the above-mentioned labeled data can be at least one item in the link fault location model or the link fault location rule base. The link fault location rule base includes at least one link fault location rule.

[0115] The following describes the training process of the link fault location model and the link fault location rule.

[0116] The link fault location model can be a classification model obtained by training a classifier in a supervised learning manner. The classifier can be a random forest classifier (RFC) or a neural network (NN) classifier. Figure 4 As shown, the link fault location system 10 can obtain training data, such as obtaining the fault log of the hardware system in the computing system deployed in the production environment. Since the computing system in the production environment can have multiple faults in a period of time, multiple existing network problem cases can be obtained. For each existing network problem case, the link fault location system 10 can extract the fault feature sequence and mark the faulty FRU, for example, marking the faulty FRU according to the FRU replacement record, identifying and marking the faulty FRU through the fault-related information recorded in the fault log, or marking the faulty FRU based on experience, marking the faulty FRU through detection means (such as instruments and meter detection), so that a case feature matrix (i.e., a fault feature sequence marked with a faulty FRU) can be obtained. The link fault location system 10 can use the case feature matrix to train a classifier, and when the training stop condition is met, the trained classifier can be determined as a link fault location model.

[0117] The link fault location rule can be obtained by matching the fault feature sequence of the faulty FRU with the rule template. The rule template can be formed by a conditional statement (such as an if-then statement), and accordingly, the link fault location rule can be a conditional statement filled with the fault feature sequence and the faulty FRU in the rule template.

[0118] In some possible implementations, the training result may be a link fault location model. Accordingly, the link fault location system 10 may input the fault feature sequence of the hardware system into the link fault location model, and obtain the fault location at the FRU granularity through the link fault location model classification. Among them, when the link fault location system 10 is classified through the link fault location model, it may determine the fault probability of the FRU in the hardware system according to the fault feature sequence of the hardware system through the link fault location model, and obtain the fault location at the FRU granularity based on the fault probability of each FRU.

[0119] Specifically, when there is a FRU with a failure probability greater than a set threshold, the link fault location system 10 can determine that the fault location includes the FRU with a failure probability greater than the set threshold. When the failure probability of the FRUs in the hardware system is not greater than the set threshold, the link fault location system 10 can determine that the fault location is one or more of the first n FRUs in order of failure probability from high to low. Furthermore, the link fault location system 10 can also output the failure probability of the FRU (fault source, suspected fault source).

[0120] In some other possible implementations, the link fault location system 10 may match the fault feature sequence of the hardware system with the link fault location rules in the link fault location rule base. The link fault location system 10 may determine the fault location at the FRU granularity based on the matching result. For example, when the matching result is a successful match, the link fault location system 10 determines the fault location as the FRU that has successfully matched. Specifically, when the fault feature sequence of the hardware system successfully matches the target rule in at least one link fault location rule, the link fault location system 10 may determine the fault location at the FRU granularity from the target rule.

[0121] Considering that the link fault location rule base includes link fault location rules that can directly determine the source of the fault, the link fault location system 10 can first match the fault feature sequence of the hardware system with the link fault location rules in the fault location rule base, for example, by performing an accurate match. When the match is successful, the link fault location system 10 can determine that the fault location is the FRU that has been successfully matched. When the match is unsuccessful, the link fault location system 10 can input the fault feature sequence of the hardware system into the link fault location model, and obtain the fault location of the FRU granularity in the hardware system through the link fault location model inference.

[0122] Based on the above description, it can be seen that this method extracts the fault feature sequence from the fault log when the fault occurs and the current data of the fault monitoring parameters, and then uses the training results (such as the link fault location model or the link fault location rule base) based on the labeled data (such as the fault feature sequence of the faulty FRU) to infer the FRU corresponding to the fault feature sequence of the current fault, thereby realizing the link fault location at the FRU level and improving the location accuracy. Moreover, this method is not limited to the location of PCIe link faults, and can realize general link fault location, thereby improving the coverage of link fault location. In addition, this method uses out-of-band collected information as the positioning input, which is easy to obtain and does not increase the operation of maintenance personnel.

[0123] Furthermore, the link fault location model or the link fault location rule base is constructed based on the fault logs of the computing systems with different link topologies in the production environment. Based on this, in the reasoning stage, the link fault location system 10 can obtain the link topology of the hardware system of the computing system to be detected, and predict the fault location of the FRU granularity based on the link topology and the fault feature sequence. For example, the link fault location system 10 can input the link topology of the hardware system of the computing system to be detected into the link fault location model, and the link fault location model is inferred based on the link topology and the fault feature sequence to obtain the fault location of the FRU granularity. For another example, the link fault location system 10 can determine the link fault location rules related to the link topology of the hardware system to be detected from the link fault location rule base, match the fault feature sequence with the link fault location rules related to the link topology of the hardware system to be detected, and obtain the fault location of the FRU granularity. In this way, the accuracy and efficiency of link fault location can be improved.

[0124] like Figure 5 As shown, the link fault location system 10 can generate a link topology of the hardware system based on the networking information of the hardware system, and the link topology can record the link type of the link in the hardware system and the FRUs included in the link of at least one link type. Accordingly, when determining the fault location at the FRU granularity, the link fault location system 10 can determine the fault location at the FRU granularity based on the fault feature sequence of the hardware system and the link topology of the hardware system through a link fault location model or a link fault location rule base.

[0125] Among them, the link fault location system 10 can determine the link fault location rules related to the links in the link topology from the link fault location rule library according to the links included in the link topology of the hardware system, and then match the fault feature sequence with the link fault location rules related to the links in the link topology, thereby determining the fault location of the FRU granularity. In some examples, the link fault location system 10 can also use the link topology of the hardware system as a priori features and input it into the link fault location model. The link fault location model can perform reasoning based on the priori features and the input fault feature sequence to determine the fault location of the FRU granularity.

[0126] Next, the process of generating a link topology is described.

[0127] In one implementation, the link fault location system 10 may generate a link topology according to hardware connection relationships. Figure 6 The link topology diagrams of different types of computing systems are shown, for example, a computing system using a SATA / SAS hard disk without an Expander, a computing system using a SATA / SAS hard disk with an Expander, a computing system using a Non-Volatile Memory Express (NVME) pass-through disk, and a computing system using NVME with a PCIE SW.

[0128] The link topology can also be represented by a link topology table, as shown below:

[0129] Table 2 Link topology table

[0130]

[0131] Among them, NA in Table 2 means Not applicable, that is, under this link type, this FRU is empty.

[0132] In another implementation, the link fault location system 10 may also number all FRUs, and then generate a link topology map or a link topology table according to the numbered FRUs. For ease of description, the generation of a link topology table is used as an example for explanation.

[0133] In this example, the numbers (also called identifiers, IDs) of all FRUs can be found in the following table:

[0134] Table 3 FRU numbering results

[0135]

[0136]

[0137] Then, the link fault location system 10 generates the following link topology table according to the hardware connection relationship:

[0138] Table 4 Link topology table

[0139]

[0140]

[0141]

[0142] It should be noted that the above method for generating link topology can be applied to hardware systems with flexible component networking, and can also be applied to hardware systems with fixed component networking, and has high availability.

[0143] The aforementioned embodiments of the present application can be implemented individually or in any combination to solve the problem that it is difficult to accurately locate the source of the fault at one time when link failures occur on various high-speed links and low-speed links during the operation of a computing system (such as a server). It can more accurately identify the link where the fault occurs and diagnose the faulty FRU (i.e., faulty FRU) or the risk FRU suspected of having a fault on the link, and can issue an alarm for the faulty FRU or the Top N risk FRUs.

[0144] Based on the above method embodiments, the present application also provides a link fault location system 10. The link fault location system 10 of the present application is described below in conjunction with the accompanying drawings.

[0145] See also Figure 2 The structure diagram of a link fault location system 10 is shown, and the link fault location system 10 includes:

[0146] The parameter and log collection module 102 is used to obtain the fault log of the hardware system in the computing system to be detected, wherein the fault log records the historical data of the fault monitoring parameters of the hardware system, abnormal events and the replacement records of the field replaceable units FRU in the hardware system, and obtains the current data of the fault monitoring parameters of the hardware system;

[0147] A preprocessing module 104, configured to extract a fault feature sequence of the hardware system according to the fault log and current data of the fault monitoring parameters of the hardware system;

[0148] The link fault location module 106 is used to predict the fault location of FRU granularity according to the fault feature sequence of the hardware system, wherein the fault location of FRU granularity is predicted in combination with the training result based on the labeled data, and the labeled data is labeled with the faulty FRU.

[0149] The specific implementation of the parameter and log collection module 102 can be found in Figure 3 The description of S302 and S304 in the embodiment shown in the figure and the specific implementation of the preprocessing module 104 can be found in Figure 3 The description of S306 in the embodiment shown in the figure and the specific implementation of the link fault location module 106 can be found in Figure 3 Description of the relevant contents of S308 in the illustrated embodiment.

[0150] In some possible implementations, the training result includes a link fault location model;

[0151] The link fault locating module 106 is specifically used for:

[0152] Inputting the fault feature sequence of the hardware system into the link fault location model to obtain the failure probability of the FRU in the hardware system;

[0153] According to the failure probability of the FRU in the hardware system, the failure location of the FRU granularity is determined.

[0154] In some possible implementations, the link fault locating module 106 is specifically configured to:

[0155] When there is a FRU with a failure probability greater than a set threshold, it is determined that the fault location includes the FRU whose failure probability is greater than the set threshold; when the failure probability of all FRUs in the hardware system is not greater than the set threshold, it is determined that the fault location is one or more of the first n FRUs sorted from high to low in failure probability.

[0156] In some possible implementations, the link fault location system 10 further includes:

[0157] A construction module 109 is used to obtain training data, wherein the training data includes a fault log of a hardware system in a computing system deployed in a production environment, extract a fault feature sequence from the training data, and annotate the faulty FRU according to the fault feature sequence to obtain the annotated data, train a classifier based on the annotated data, and obtain the link fault location model.

[0158] The specific implementation of building module 109 to train the link fault location model can be found in Figure 4 The relevant content description will not be repeated here.

[0159] In some possible implementations, the training result includes a link fault location rule base, and the link fault location rule base includes at least one link fault location rule;

[0160] The link fault locating module 106 is specifically used for:

[0161] Matching the fault feature sequence of the hardware system with the at least one link fault locating rule;

[0162] When the fault feature sequence of the hardware system successfully matches the target rule in the at least one link fault location rule, the fault location at the FRU granularity is determined from the target rule.

[0163] In some possible implementations, the training result includes a link fault location model and a link fault location rule base, and the link fault location rule base includes at least one link fault location rule;

[0164] The link fault locating module is specifically used for:

[0165] Matching the fault feature sequence of the hardware system with the at least one link fault locating rule;

[0166] When the matching fails, the fault feature sequence of the hardware system is input into the link fault location model to obtain the fault location at the FRU granularity.

[0167] The specific implementation of the link fault location module 106 performing FRU-level fault location based on the link fault location rule and the link fault location model can be found in Figure 5 Description of related content.

[0168] In some possible implementations, the preprocessing module 104 is further configured to:

[0169] Generate a link topology of the hardware system according to the networking information of the hardware system, wherein the link topology records link types of links in the hardware system and FRUs included in links of at least one link type;

[0170] The link fault locating module 106 is specifically used for:

[0171] The fault location at FRU granularity is predicted according to the fault feature sequence of the hardware system and the link topology of the hardware system.

[0172] In some possible implementations, the fault monitoring parameters include one or more of temperature, current, voltage, or in-position state and plugged-in state; the abnormal events include one or more of sideband detection abnormality, lighting abnormality, reliability availability, serviceability (RAS) abnormality; and the RAS abnormality includes data access abnormality during the operation of a chip, controller, bus, or input / output (IO) peripheral.

[0173] In some possible implementations, the preprocessing module 104 is specifically used to:

[0174] Extract one or more of the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event from the fault log;

[0175] The fault feature sequence of the hardware system is constructed according to one or more of the number of occurrences, the occurrence time, the event subject or the link where the event is located, and the historical data and current data of the fault monitoring parameters of the abnormal event.

[0176] Based on the aforementioned link fault location method and link fault location system 10, the present application also provides a baseboard management controller, also called a motherboard management control unit. The baseboard management controller includes a processor and a memory, and the memory can store computer-readable instructions, such as program instructions of the link fault location system 10. The processor executes the above-mentioned computer-readable instructions so that the baseboard management controller executes the aforementioned link fault location method.

[0177] Based on the aforementioned link fault location method and link fault location system 70, the present application also provides a computing device 700. Figure 7 As shown, computing device 700 includes: bus 702, processor 704, memory 706 and communication interface 708. Processor 704, memory 706 and communication interface 708 communicate through bus 702. Computing device 700 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 700.

[0178] The bus 702 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 702 may include a path for transmitting information between various components of the computing device 700 (eg, the memory 706, the processor 704, and the communication interface 708).

[0179] The processor 704 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0180] The memory 706 may include a volatile memory, such as a random access memory (RAM). The memory 706 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 706 stores executable program code, and the processor 704 executes the executable program code to implement the aforementioned link fault location method. Specifically, the memory 706 stores instructions for the link fault location system 10 to execute the link fault location method.

[0181] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or communication networks.

[0182] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0183] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same link fault location system 10 for executing the link fault location method.

[0184] In some possible implementations, one or more computing devices 700 in the computing device cluster may also be used to execute some instructions of the link fault location system 10 for executing the link fault location method. In other words, a combination of one or more computing devices 700 may jointly execute instructions of the link fault location system 10 for executing the link fault location method.

[0185] It should be noted that the memory 706 in different computing devices 700 in the computing device cluster may store different instructions for executing part of the functions of the link fault location system 10 .

[0186] Fig. 9 A possible implementation is shown. Fig. 9As shown, two computing devices 700A and 700B are connected via a communication interface 708. The memory in the computing device 700A stores instructions for executing the functions of the parameter and log collection module 102 and the preprocessing module 104. The memory in the computing device 700B stores instructions for executing the functions of the link fault location module 106. In other words, the memories 706 of the computing devices 700A and 700B jointly store instructions for the link fault location system 10 to execute the link fault location method.

[0187] Fig. 9 The connection mode between the computing device clusters shown can be considered that the link fault location method provided by the present application requires a lot of computing power for reasoning or matching, so as to predict the fault location of the FRU granularity. Therefore, it is considered that the functions implemented by the link fault location module 106 are handed over to the computing device 700B for execution. Furthermore, the functions of the output module 108 and the construction module 109 can also be handed over to the computing device 700B for execution.

[0188] It should be understood that Fig. 9 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices 700.

[0189] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 700C and 700D are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 706 in the computing device 700C stores instructions for executing the functions of the parameter and log collection module 102 and the preprocessing module 104. At the same time, the memory 706 in the computing device 700D stores instructions for executing the functions of the link fault location module 106.

[0190] Fig.10 The connection mode between the computing device clusters shown may be considered to be that the link fault location method provided by the present application requires a large amount of computing power for reasoning or matching, so as to predict the fault location of FRU granularity. Therefore, it is considered to hand over the functions implemented by the link fault location module 106 to the computing device 700D for execution.

[0191] It should be understood that Fig.10 The functions of the computing device 700C shown in FIG. 700A may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700D may also be completed by multiple computing devices 700.

[0192] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned link fault location system 10 for executing the link fault location method.

[0193] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned link fault location method.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A link fault location method, It is characterized in that The method comprises: Obtaining a fault log of a hardware system in the computing system to be detected, wherein the fault log records historical data of fault monitoring parameters of the hardware system, abnormal events, and replacement records of field replaceable units (FRUs) in the hardware system, and obtaining current data of the fault monitoring parameters of the hardware system; Extracting a fault feature sequence of the hardware system according to the fault log and current data of the fault monitoring parameters of the hardware system; The fault location of the FRU granularity is predicted according to the fault feature sequence of the hardware system, wherein the fault location of the FRU granularity is predicted in combination with the training result based on the labeled data, and the labeled data is labeled with the faulty FRU.

2. The method according to claim 1, It is characterized in that The training results include a link fault location model; The predicting of the fault location at FRU granularity according to the fault feature sequence of the hardware system includes: Inputting the fault feature sequence of the hardware system into the link fault location model to obtain the failure probability of the FRU in the hardware system; According to the failure probability of the FRU in the hardware system, the failure location of the FRU granularity is determined.

3. The method according to claim 2, It is characterized in that Determining the fault location of the FRU granularity according to the fault probability of the FRU in the hardware system includes: When there is a FRU with a failure probability greater than a set threshold, it is determined that the fault location includes the FRU whose failure probability is greater than the set threshold; when the failure probability of all FRUs in the hardware system is not greater than the set threshold, it is determined that the fault location is one or more of the first n FRUs sorted from high to low in failure probability.

4. The method according to claim 2 or 3, It is characterized in that The link fault location model is constructed in the following way: Acquire training data, where the training data includes a fault log of a hardware system in a computing system deployed in a production environment; Extracting a fault feature sequence from the training data, and marking the faulty FRU according to the fault feature sequence to obtain the marked data; A classifier is trained according to the labeled data to obtain the link fault location model.

5. The method according to claim 1, It is characterized in that The training result includes a link fault location rule base, and the link fault location rule base includes at least one link fault location rule; The predicting of the fault location at FRU granularity according to the fault feature sequence of the hardware system includes: Matching the fault feature sequence of the hardware system with the at least one link fault locating rule; When the fault feature sequence of the hardware system successfully matches the target rule in the at least one link fault location rule, the fault location at the FRU granularity is determined from the target rule.

6. The method according to claim 1, It is characterized in that The training result includes a link fault location model and a link fault location rule base, wherein the link fault location rule base includes at least one link fault location rule; The predicting of the fault location at FRU granularity according to the fault feature sequence of the hardware system includes: Matching the fault feature sequence of the hardware system with the at least one link fault locating rule; When the matching fails, the fault feature sequence of the hardware system is input into the link fault location model to obtain the fault location at the FRU granularity.

7. The method according to any one of claims 1 to 6, It is characterized in that The method further comprises: Generate a link topology of the hardware system according to the networking information of the hardware system, wherein the link topology records link types of links in the hardware system and FRUs included in links of at least one link type; The predicting of the fault location at FRU granularity according to the fault feature sequence of the hardware system includes: The fault location at FRU granularity is predicted according to the fault feature sequence of the hardware system and the link topology of the hardware system.

8. The method according to any one of claims 1 to 7, It is characterized in that The fault monitoring parameters include one or more of temperature, current, voltage, in-position state, and plugged-in state; the abnormal events include one or more of sideband detection abnormality, lighting abnormality, and reliability, availability, and serviceability (RAS) abnormality; the RAS abnormality includes data access abnormality during the operation of a chip, controller, bus, or input / output (IO) peripheral.

9. The method according to any one of claims 1 to 8, It is characterized in that The extracting the fault feature sequence of the hardware system according to the fault log and the current data of the fault monitoring parameters of the hardware system comprises: Extract one or more of the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event from the fault log; The fault feature sequence of the hardware system is constructed according to one or more of the number of occurrences, the occurrence time, the event subject or the link where the event is located, and the historical data and current data of the fault monitoring parameters of the abnormal event.

10. A link fault location system, It is characterized in that The system comprises: A parameter and log collection module, used to obtain a fault log of a hardware system in a computing system to be detected, wherein the fault log records historical data of fault monitoring parameters of the hardware system, abnormal events, and replacement records of field replaceable units (FRUs) in the hardware system, and obtains current data of fault monitoring parameters of the hardware system; A preprocessing module, used for extracting a fault feature sequence of the hardware system according to the fault log and current data of the fault monitoring parameters of the hardware system; A link fault location module is used to predict the fault location of FRU granularity according to the fault feature sequence of the hardware system, wherein the fault location of FRU granularity is predicted in combination with the training result based on the labeled data, and the labeled data is labeled with the faulty FRU.

11. The system according to claim 10, It is characterized in that The training results include a link fault location model; The link fault locating module is specifically used for: Inputting the fault feature sequence of the hardware system into the link fault location model to obtain the failure probability of the FRU in the hardware system; According to the failure probability of the FRU in the hardware system, the failure location of the FRU granularity is determined.

12. The system according to claim 11, It is characterized in that The link fault locating module is specifically used for: When there is a FRU with a failure probability greater than a set threshold, it is determined that the fault location includes the FRU whose failure probability is greater than the set threshold; when the failure probability of all FRUs in the hardware system is not greater than the set threshold, it is determined that the fault location is one or more of the first n FRUs sorted from high to low in failure probability.

13. The system according to claim 11 or 12, It is characterized in that The link fault locating system further includes: A construction module is used to obtain training data, wherein the training data includes fault logs of hardware systems in a computing system deployed in a production environment, extract fault feature sequences from the training data, and mark faulty FRUs according to the fault feature sequences to obtain the marked data, train a classifier based on the marked data, and obtain the link fault location model.

14. The system according to claim 10, It is characterized in that The training result includes a link fault location rule base, and the link fault location rule base includes at least one link fault location rule; The link fault locating module is specifically used for: Matching the fault feature sequence of the hardware system with the at least one link fault locating rule; When the fault feature sequence of the hardware system successfully matches the target rule in the at least one link fault location rule, the fault location at the FRU granularity is determined from the target rule.

15. The system according to claim 10, It is characterized in that The training result includes a link fault location model and a link fault location rule base, wherein the link fault location rule base includes at least one link fault location rule; The link fault locating module is specifically used for: Matching the fault feature sequence of the hardware system with the at least one link fault locating rule; When the matching fails, the fault feature sequence of the hardware system is input into the link fault location model to obtain the fault location at the FRU granularity.

16. A system according to any one of claims 10 to 15, It is characterized in that The preprocessing module is also used for: Generate a link topology of the hardware system according to the networking information of the hardware system, wherein the link topology records link types of links in the hardware system and FRUs included in links of at least one link type; The link fault locating module is specifically used for: The fault location at FRU granularity is predicted according to the fault feature sequence of the hardware system and the link topology of the hardware system.

17. A system according to any one of claims 10 to 16, It is characterized in that The fault monitoring parameters include one or more of temperature, current, voltage, in-position state, and plugged-in state; the abnormal events include one or more of sideband detection abnormality, lighting abnormality, and reliability, availability, and serviceability (RAS) abnormality; the RAS abnormality includes data access abnormality during the operation of a chip, controller, bus, or input / output (IO) peripheral.

18. A system according to any one of claims 10 to 17, It is characterized in that The preprocessing module is specifically used for: Extract one or more of the number of occurrences, the time of occurrence, the event subject, or the link where the event is located of the abnormal event from the fault log; The fault feature sequence of the hardware system is constructed according to one or more of the number of occurrences, the occurrence time, the event subject or the link where the event is located, and the historical data and current data of the fault monitoring parameters of the abnormal event.

19. A baseboard management controller, It is characterized in that The baseboard management controller comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the baseboard management controller executes the method according to any one of claims 1 to 9.

20. A computing device cluster, It is characterized in that The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 9.

21. A computer-readable storage medium, It is characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images