A troubleshooting method, device and storage medium
By responding to temperature alarm logs, the target components and servers are identified, and sensor and temperature information is obtained for programmed fault diagnosis. This solves the problems of low efficiency and insufficient accuracy caused by manual reliance in existing technologies, and realizes automated and efficient fault diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT TECHNOLOGY (SHENZHEN) CO LTD
- Filing Date
- 2021-08-31
- Publication Date
- 2026-04-28
AI Technical Summary
In existing technologies, server fault diagnosis relies on manual judgment, which is inefficient and prone to information omissions, and cannot provide timely and accurate fault diagnosis solutions.
By responding to temperature alarm logs, the target components and servers are identified, and sensor information of the target components and suspected components, as well as temperature information of the deployment units, is obtained. Based on this information, a programmed troubleshooting process is performed.
It has achieved automated fault diagnosis, reduced manual interference, improved fault diagnosis efficiency, reduced labor costs, and ensured the accuracy of the diagnosis.
Smart Images

Figure CN115934453B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Internet communication technology, and in particular to a fault diagnosis method, apparatus and storage medium. Background Technology
[0002] Server component malfunctions often affect the server's ability to support related business functions. In practice, staff typically identify the malfunctioning server component by combining various logs and information about the business system's functionality, and then perform troubleshooting. This process relies heavily on individual judgment and lacks a structured, procedural troubleshooting logic. This excessive reliance on manual labor often leads to inefficiency, information omissions, and ultimately, fails to provide timely and accurate troubleshooting solutions. Summary of the Invention
[0003] To address the shortcomings of existing technologies in fault diagnosis, such as their inability to provide timely and accurate fault diagnosis solutions, this application provides a fault diagnosis method, apparatus, and storage medium:
[0004] According to a first aspect of this application, a fault diagnosis method is provided, the method comprising:
[0005] In response to temperature alarm logs for a specified server cluster, determine the target component and target server indicated by the temperature alarm logs; wherein, the target component is the server component in the target server that has generated a temperature alarm, and the target server is any server in the specified server cluster;
[0006] Acquire the first type of sensor information of the target component and the second type of sensor information of the suspicious component; wherein, the suspicious component is a server component in the target server that is suspected of being associated with the temperature alarm triggered by the target component;
[0007] Obtain the temperature information of the deployment unit to which the target server belongs;
[0008] Based on the information from the first type of sensor, the information from the second type of sensor, and the temperature information, the troubleshooting results for the temperature alarm log are determined.
[0009] According to a second aspect of this application, a fault diagnosis device is provided, the device comprising:
[0010] First determination module: used to determine the target component and target server indicated by the temperature alarm log in response to the temperature alarm log for a specified server cluster; wherein, the target component is the server component in the target server that has a temperature alarm, and the target server is any server in the specified server cluster;
[0011] First acquisition module: used to acquire first type of sensor information of the target component and second type of sensor information of the suspicious component; wherein, the suspicious component is a server component in the target server that has a suspected correlation with the temperature alarm triggered by the target component;
[0012] The second acquisition module is used to acquire the temperature information of the deployment unit to which the target server belongs.
[0013] The second determining module is used to determine the troubleshooting results for the temperature alarm log based on the information from the first type of sensor, the information from the second type of sensor, and the temperature information.
[0014] According to a third aspect of this application, an electronic device is provided, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the troubleshooting method as described in the first aspect.
[0015] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the troubleshooting method as described in the first aspect.
[0016] According to a fifth aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the troubleshooting method as described in the first aspect.
[0017] The fault diagnosis method, apparatus, and storage medium provided in this application have the following technical advantages:
[0018] This application identifies the target component and target server indicated by the temperature alarm log, then obtains the first type of sensor information of the target component, the second type of sensor information of the suspected component, and the temperature information of the deployment unit to which the target server belongs. Based on the obtained information, it determines the troubleshooting results for the temperature alarm log. The target component is the server component in the target server that triggered the temperature alarm, and the suspected component is a server component in the target server that is suspected of having a correlation with the component that triggered the temperature alarm. This application provides a procedural troubleshooting scheme that avoids interference from personal subjective judgment, reduces missed and false alarms, and thus enables accurate troubleshooting. The reduction of manual intervention and the automation of the entire troubleshooting process improves troubleshooting efficiency and saves labor costs. Attached Figure Description
[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;
[0021] Figure 2 This is a flowchart illustrating a troubleshooting method provided in an embodiment of this application;
[0022] Figure 3 This is a flowchart illustrating a method for determining the troubleshooting results for temperature alarm logs, as provided in an embodiment of this application.
[0023] Figure 4 This is a schematic flowchart illustrating the process for handling the number of false alarms provided in an embodiment of this application;
[0024] Figure 5 This is a schematic diagram of the temperature monitoring log provided in an embodiment of this application;
[0025] Figure 6 This is also a flowchart illustrating a troubleshooting method provided in an embodiment of this application;
[0026] Figure 7 This is a schematic diagram of the out-of-band management system provided in an embodiment of this application;
[0027] Figure 8 This is a schematic diagram of the hardware replacement process of the server fault handling system provided in the embodiments of this application;
[0028] Figure 9 This is also a flowchart illustrating a troubleshooting method provided in an embodiment of this application;
[0029] Figure 10 This is a block diagram of a fault diagnosis device provided in an embodiment of this application;
[0030] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0032] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.
[0033] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0034] BMC (Baseboard Management Controller): Baseboard management controller.
[0035] SDR (Sensor Data Record): Sensor data recording.
[0036] SNMPTRAP (Simple Network Management Protocol TRAP): Based on the Simple Network Management Protocol trap format.
[0037] IPMI (Intelligent Platform Management Interface): Intelligent Platform Management Interface.
[0038] Please see Figure 1 , Figure 1This is a schematic diagram of an application environment provided in an embodiment of this application. This application environment may include a designated server cluster 10 and a server 20. The designated server cluster 10 and the server 20 can be directly or indirectly connected via wired or wireless communication. In response to a temperature alarm log for the designated server cluster, the server 20 determines the target component and target server indicated by the temperature alarm log. Then, it acquires the first type of sensor information of the target component, the second type of sensor information of the suspected component, and the temperature information of the deployment unit to which the target server belongs. Based on the acquired information, it determines the fault diagnosis result for the temperature alarm log, thereby achieving fault diagnosis for the designated server cluster 10. It can be understood that the designated server cluster 10 is a server cluster monitored by the server 20. It should be noted that... Figure 1 This is just one example.
[0039] Server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc. The server can provide backend services to the corresponding clients.
[0040] The troubleshooting method provided in this application embodiment can be implemented based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.
[0041] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go. As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed within this pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices. Logically, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be directly deployed on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally, SaaS and PaaS are upper layers compared to IaaS.
[0042] In practical applications, a designated server cluster 10 can consist of multiple physical servers, which can be located in different deployment units. Deployment units can be racks, data centers, etc. For example, the designated server cluster includes server cluster ac, and each server cluster includes multiple physical servers. Server cluster a is located in deployment unit a, server cluster b is located in deployment unit b, and server cluster c is located in deployment unit c. The server-side 20 can include an out-of-band management system, a server hardware fault diagnosis system, a server fault handling system, and an online diagnostic system.
[0043] The following describes a specific embodiment of a fault diagnosis method according to this application. Figure 2 This is a flowchart illustrating a troubleshooting method provided in an embodiment of this application. This application provides the operational steps of the method described in the embodiment or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiment is merely one possible execution order among many and does not represent the only possible execution order. In actual system or product execution, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiment or drawings. Specifically, as shown... Figure 2As shown, the method may include:
[0044] S201: In response to a temperature alarm log for a specified server cluster, determine the target component and target server indicated by the temperature alarm log; wherein, the target component is the server component in the target server that has a temperature alarm, and the target server is any server in the specified server cluster.
[0045] In this embodiment, the server receives temperature alarm logs for a specified server cluster. The server then determines the target component and target server indicated by the temperature alarm logs. The temperature alarm logs indicate that a certain server component within a server in the specified server cluster has triggered a temperature alarm, meaning the real-time temperature of that server component is higher than its corresponding temperature threshold. The temperature alarm logs are server-level log information, containing not only the real-time information of the server component that triggered the temperature alarm but also the real-time information of other server components within the same server. Based on the temperature alarm logs, the server determines the target component that triggered the temperature alarm and the target server to which that target component belongs.
[0046] Server components can include the CPU (Central Processing Unit), fan assembly, air intake assembly, air exhaust assembly, memory, RAID card (RAID array), PCH (integrated southbridge), etc. Correspondingly, each server component has a different temperature threshold. Different temperature parameters and corresponding temperature thresholds can also be set for the same server component.
[0047] It is understandable that temperature monitoring of server components within a specified server cluster is real-time, and this real-time information can generate corresponding temperature monitoring logs based on the server level (see [link to relevant documentation]). Figure 5 Temperature monitoring logs can be stored for reference. Logs showing real-time temperatures exceeding corresponding temperature thresholds are designated as temperature alarm logs, which can serve as data sources for triggering troubleshooting. Setting up temperature alarm logs provides effective evidence for troubleshooting and narrows down the data scope for investigation.
[0048] In one exemplary embodiment, before determining the target component indicated by the temperature alarm log and the target server, the method further includes: receiving the temperature alarm log; wherein the temperature alarm log is a log based on Simple Network Management Protocol Trap Format (SNMPTRAP);
[0049] Accordingly, determining the target component indicated by the temperature alarm log and the target server includes the following steps: 1) parsing the temperature alarm message to obtain a first field indicating server identification and a second field indicating component temperature alarm; 2.1) determining the target server based on the first field; 2.2) matching the second field with a set of reference fields indicating component temperature alarm, and when there is a reference field in the set of reference fields that matches the second field, determining the target component based on the component indicated by the reference field.
[0050] The BMC of the server to which the server component that triggered the temperature alarm belongs reports the temperature alarm log to the server based on the real-time information of the server component within that server. The temperature alarm log is a trap message reported by the manager (agent) based on the Simple Network Management Protocol (SMMP), used to proactively notify the device of a failure.
[0051] Temperature alarm logs are server-level log information. Since different servers may have differences in model, provider, etc., and temperature alarm logs originate from raw logs, even if the temperature alarm logs are in SNMPTRAP format, different model and different provider will still cause differences in temperature alarm logs.
[0052] The process of the server parsing temperature alarm logs can be viewed as obtaining standard output. During this process, a pre-defined mapping table can be used to standardize personalized fields in the temperature alarm logs that are configured differently depending on the device model and the vendor. For example, in temperature alarm logs indicating vendor A, the temperature unit is Fahrenheit, which is standardized to Celsius. In temperature alarm logs indicating device B, the field indicating the server identifier is placed later, and after standardization, this field is placed earlier. Standardization improves the efficiency of locating relevant fields.
[0053] The parsing process yields a first field indicating the server identifier and a second field indicating the component temperature alarm. Based on the first field, the target server is determined, and based on the second field, the target component is determined. Specifically, the second field can be matched against a set of reference fields indicating the component temperature alarm. When a reference field matching the second field exists in the set, the target component is determined based on the component indicated by that reference field. The set of reference fields may include fields indicating the CPU, fields indicating the air inlet assembly, fields indicating the air outlet assembly, fields indicating memory, etc. If the second field matches a field indicating memory, then the memory component can be identified as the target component based on that memory-indicating field.
[0054] In practical applications, the out-of-band management system on the server side receives temperature alarm logs and performs log parsing and field matching. See also Figure 6 , 7 The out-of-band management system's access layer receives raw SNMPTRAP logs from the existing network. A designated server cluster can contain millions of servers, each generating a large amount of log information daily. To receive SNMPTRAP logs from all servers in the cluster, corresponding access layer devices can be deployed on each cluster to receive and initially process the large volume of raw logs. The out-of-band management system's parsing layer parses the raw SNMPTRAP logs to improve the efficiency of subsequent data application. The out-of-band management system's alarm matching layer matches component temperature alarm data. After reception by the access layer and parsing by the parsing layer, the data in the raw SNMPTRAP logs is ready for further processing, allowing the system to determine which machine experienced what type of fault.
[0055] In an exemplary embodiment, after determining the target component indicated by the temperature alarm log and the target server, the method further includes the following steps: 1) determining the instruction issuance time based on the log reception time and a preset time interval; wherein, the log reception time indicates the time when the temperature alarm log is received; 2.1) at the instruction issuance time, sending a first instruction for obtaining the first type of sensor information and the second type of sensor information to the sensor information management module; wherein, the sensor information management module is used to manage the sensor information obtained by monitoring the server component in the target server; 2.2) at the instruction issuance time, sending a second instruction for obtaining the temperature information to the temperature management system corresponding to the deployment unit.
[0056] By setting a preset time interval, a buffer period is created between the acquisition of relevant sensor and temperature information and the current log reception time. This buffer period allows the required sensor and temperature information to converge, ensuring that it serves as more accurate characterization data. This provides more accurate and effective reference data for subsequent troubleshooting, thereby guaranteeing the accuracy of the troubleshooting process.
[0057] For example, if the log reception time is 17:00 and the preset time interval is 5 minutes, then the instruction issuance time is 17:05. At this specified issuance time, a first instruction for obtaining first-type sensor information and second-type sensor information is sent to the sensor information management module, and a second instruction for obtaining temperature information is sent to the temperature management system. The content regarding the first-type sensor information, the second-type sensor information, and the temperature information will be described in steps S202 and S203 below. In practical applications, see [link to relevant documentation]. Figure 6 The sensor information management module can be the BMC of the target server. The first and second commands can be in the form of IPMI commands. Of course, the preset time interval can be flexibly set as needed.
[0058] Furthermore, monitoring logs (such as temperature monitoring logs) can be retrieved at the time the command is issued. These logs can be from the target server or from all servers within the deployment unit to which the target server belongs. Monitoring logs can broaden the scope of subsequent troubleshooting, preventing information omissions from affecting the accuracy of troubleshooting results.
[0059] S202: Obtain the first type of sensor information of the target component and the second type of sensor information of the suspicious component; wherein, the suspicious component is a server component in the target server that is suspected of being associated with the temperature alarm triggered by the target component;
[0060] In this embodiment, the server acquires first-type sensor information of the target component and second-type sensor information of suspicious components. The reason for a temperature alarm in the target component may be due to its own problem, a problem with other server components, excessively high ambient temperature, inaccurate real-time temperature data in the temperature alarm log (e.g., a faulty component detecting the temperature of the target component), or an excessively low temperature threshold, etc. Suspicious components correspond to the aforementioned "problems with other server components"; they are server components on the target server that are suspected of having a correlation with the cause of the temperature alarm in the target component.
[0061] The first type of sensor information is the SDR (Survey Reporting Item) obtained by detecting the target component using relevant sensors. The second type of sensor information is the SDR obtained by detecting suspicious components using relevant sensors. Since the relevant sensors may detect more than one detection item for the target component, the first type of sensor information can extract the SDRs indicating temperature-related detection items from all SDRs. The second type of sensor information is obtained similarly.
[0062] The temperature alarm log contains not only real-time information about the server component that triggered the temperature alarm, but also real-time information about other server components within the same server. The real-time information about the server components in the temperature alarm log, or temperature monitoring log, can also come from the SDR (Special Data Retrieval) of the relevant sensors. Obtaining the first and second type of sensor information here can be understood as retrieving the first and second type of sensor information from the temperature alarm log. First, the temperature alarm log is parsed, and then the sensor field is determined based on the parsed data. At this point, the first type of sensor information can serve as all or part of the data source that triggers the conversion of ordinary temperature monitoring logs into temperature alarm logs, or it can serve as reference data for subsequent troubleshooting. Combined with the "first instruction" mentioned in step S201 above, the first type of sensor information, the second type of sensor information, and the temperature alarm log point to different points in time.
[0063] S203: Obtain the temperature information of the deployment unit to which the target server belongs;
[0064] In this embodiment, the server obtains the temperature information of the deployment unit to which the target server belongs. This deployment unit can correspond to the "excessively high ambient temperature" mentioned in step S202. This temperature information can be obtained from the temperature management system corresponding to the deployment unit, which monitors the temperature of the deployment unit. For example, see [link to example]. Figure 6 The deployment unit can be a data center, and the temperature management system can be a data center temperature alarm system. When the real-time temperature of the data center (or the temperature statistics for a given time window) exceeds its corresponding temperature threshold, the data center temperature alarm system can issue a data center temperature alarm signal. A comprehensive judgment of the data center ambient temperature can be made based on the air inlet temperature of each machine on the rack and the temperature sensor values on the racks to confirm whether the current ambient temperature is within a reasonable range. Of course, the data center temperature alarm system can serve more than one data center.
[0065] The time for acquiring temperature information can be either the "log receiving time" mentioned in step S201 above, or the "instruction issuance time" mentioned in step S201 above.
[0066] S204: Based on the information from the first type of sensor, the information from the second type of sensor, and the temperature information, determine the troubleshooting results for the temperature alarm log.
[0067] In this embodiment, the server determines the troubleshooting results for the temperature alarm log based on the first type of sensor information, the second type of sensor information, and temperature information. By using the first type of sensor information, the second type of sensor information, and temperature information, it is possible to determine to some extent whether the temperature alarm of the target component is caused by its own problem, a problem with other server components, excessively high ambient temperature, or other issues.
[0068] Based on the acquisition time of relevant sensor information and temperature information in the aforementioned steps S202 and S203, two situations can be identified: 1) Acquire the first type of sensor information, the second type of sensor information, and temperature information at the "log receiving time", and then perform fault diagnosis based on this; if the fault diagnosis is not effective, acquire the "first type of sensor information, the second type of sensor information, and temperature information" after a certain buffer time to continue troubleshooting; 2) Acquire the first type of sensor information, the second type of sensor information, and temperature information at the "specified sending time", and then perform fault diagnosis based on this.
[0069] The following will describe situation 2):
[0070] See Figure 3 The step of determining the troubleshooting results for the temperature alarm log based on the first type of sensor information, the second type of sensor information, and the temperature information includes:
[0071] S301: Based on the comparison result between the first type of sensor information and the first preset quantity threshold, determine the verification result for the temperature alarm log;
[0072] S302: When the verification result indicates that the target component has a temperature alarm, based on the comparison result of the second type of sensor information and the second preset quantity threshold, and the comparison result of the temperature information and the third preset quantity threshold, determine whether there is a suspicious fault cause that causes the target component to have a temperature alarm.
[0073] S303: When there is a suspected cause of failure, obtain the first type of failure investigation result obtained by investigating and processing the suspected cause of failure.
[0074] S304: When no suspicious cause of failure exists, obtain the second type of failure investigation result obtained by investigating the target component.
[0075] The first type of sensor information, the second type of sensor information, and the temperature information here are all acquired after a certain buffer period. By comparing the first type of sensor information with the first preset quantity threshold, the verification result of the temperature alarm log can be obtained. If the temperature value indicated by the first type of sensor information is higher than the first preset quantity threshold, it means that the target component still issued a temperature alarm after a certain buffer period. The probability that the temperature alarm of the target component was caused by temperature fluctuations or false alarms is low, so further troubleshooting is required.
[0076] Based on the comparison results of the second type of sensor information and the second preset quantity threshold, it can be determined whether a suspected component has malfunctioned. Specifically, this can be obtained by comparing the values indicated by the second type of sensor information with the second preset quantity threshold. Based on the comparison results of temperature information and the third preset quantity threshold, it can be determined whether the ambient temperature is abnormal. Specifically, this can be obtained by comparing the values indicated by the temperature information with the third preset quantity threshold. A suspected component malfunction and / or abnormal ambient temperature can be considered as possible causes of the suspected malfunction.
[0077] When a suspected cause of the fault exists, troubleshooting should be conducted based on the suspected component and / or ambient temperature to obtain the first type of fault diagnosis result. For example, replace the suspected component and then check if the target component no longer emits temperature alarms; adjust the ambient temperature to a normal range and then check if the target component no longer emits temperature alarms. When no suspected cause of the fault exists, troubleshooting should be conducted based on the target component to obtain the second type of fault diagnosis result. For example, replace the old target component with a new target component and then check if the new target component emits temperature alarms.
[0078] The above embodiments provide a procedural fault diagnosis scheme. Based on unified logic, analysis and location can be performed, which can effectively improve the efficiency of fault diagnosis. The above "determine the verification result for temperature alarm logs" - "determine the cause of suspected faults" - "troubleshooting and handling" can be used as a main logical line, and logical branches can be flexibly introduced on this basis, thereby ensuring the scalability and adaptability of the main logical line.
[0079] 1) Here we will introduce a logical branch:
[0080] like Figure 4 As shown, after determining the verification result for the temperature alarm log based on the comparison result of the first type of sensor information and the first preset quantity threshold, the method further includes:
[0081] S401: When the verification result indicates that the target component has not issued a temperature alarm, update the false alarm count indicated to the target server;
[0082] S402: Determine the monitoring time corresponding to the first type of sensor information, and the monitoring period to which the monitoring time belongs;
[0083] S403: When the number of false alarms corresponding to the target server within the monitoring period is greater than the fourth preset number threshold, the steps of comparing the second type of sensor information with the second preset number threshold and comparing the temperature information with the third preset number threshold are triggered.
[0084] If the temperature value indicated by the first type of sensor is not higher than the first preset threshold, it means that the target component will no longer generate temperature alarms after a certain buffer time. The previous temperature alarms of the target component may have been caused by temperature fluctuations or false alarms. However, if such errors frequently occur within the monitoring period, it also indicates that further troubleshooting is needed. The tolerance for errors is controlled within a certain number of occurrences, taking into account both the security of the designated server cluster and the reasonable use of resources for troubleshooting.
[0085] If the monitoring period is one day and the fourth preset threshold is 2, if the number of false alarms or temperature fluctuations pointing to the target server within a day is less than or equal to 2, only this needs to be recorded, and the troubleshooting process can be ended. However, if the number of false alarms or temperature fluctuations pointing to the target server within a day is greater than 2 (i.e., 3 or more), further troubleshooting is required. See [link to relevant documentation]. Figure 9 In the process, "Node 3" → "Node 4".
[0086] 2) Here, step S302 will be explained further:
[0087] In one exemplary embodiment, the suspected component may include an air inlet assembly and a fan assembly. Accordingly, the second type of sensor information includes the temperature value of the air inlet assembly and the rotational speed value of the fan assembly, and the second preset quantity threshold includes the temperature threshold of the air inlet assembly and the rotational speed threshold of the fan assembly. Therefore, step S302 can contain the following logic for determining the suspected fault cause:
[0088] A) When the temperature value is less than or equal to the temperature threshold, determine whether the rotation speed value is less than the rotation speed threshold; if it is less, consider the fan assembly failure as the first type of suspected fault cause that triggers the temperature alarm of the target component.
[0089] See Figure 9In the sequence "Node 4" -> "Node 6" -> "Node 7", if the temperature values are less than or equal to the temperature threshold, it indicates that the inlet temperature is normal, suggesting a low probability of inlet component failure. If the inlet temperature is normal, the temperature alarm is likely caused by a cooling issue on the target server. Therefore, it's crucial to further compare the parameters of the fan components providing cooling. If the fan speed is less than the threshold, it indicates a higher probability of fan component failure. Thus, fan component failure can be considered the first suspected cause of the temperature alarm in the target component.
[0090] B) When the temperature value is greater than the temperature threshold, determine whether the ambient temperature is normal based on the comparison result between the temperature information and the third preset quantity threshold; if normal, determine whether the rotation speed value is less than the rotation speed threshold; if less, consider the fan assembly failure as the first type of suspected fault cause that triggers the temperature alarm of the target component.
[0091] See Figure 9 In the sequence "Node 4" -> "Node 5" -> "Node 6" -> "Node 7", a temperature value greater than the temperature threshold indicates an abnormal inlet air temperature. The cause of abnormal inlet air temperature is often ambient temperature, therefore it's necessary to confirm whether the ambient temperature is normal. If the ambient temperature is normal, it suggests that the relevant parameters of the fan assembly providing cooling may be abnormal, so they should be compared. A fan speed value less than the speed threshold indicates a high probability of fan assembly failure. Therefore, fan assembly failure can be considered the first suspected cause of temperature alarms in the target component.
[0092] It should be noted that in A) and B), the statement "a fan speed value less than the speed threshold indicates a higher probability of fan assembly failure" can also be interpreted by using the number of failed fans as a criterion. For example, if a fan assembly consists of 6 fans, and only one fan's speed value is below the speed threshold, then considering N+1 redundancy (meaning that theoretically, one fan's speed value below the speed threshold does not affect the fan assembly's heat dissipation), the fan assembly is not considered to be faulty.
[0093] C) When the temperature value is greater than the temperature threshold, determine whether the ambient temperature is normal based on the comparison result between the temperature information and the third preset quantity threshold; if it is not normal, take the abnormal ambient temperature as the second type of suspected fault cause that triggers the temperature alarm of the target component.
[0094] See Figure 9In the process of checking "Node 4" to "Node 5", if the temperature value exceeds the temperature threshold, it indicates an abnormal inlet air temperature. The cause of abnormal inlet air temperature is often the ambient temperature, so it is necessary to confirm whether the ambient temperature is normal. When the ambient temperature is abnormal, it can be considered as a second type of suspected fault cause for the target component to trigger a temperature alarm.
[0095] The logic described above for identifying the cause of a suspected malfunction integrates the temperature values of the air inlet assembly, the speed values of the fan assembly, and the comparison of temperature information. This provides more sufficient logical evidence for identifying the cause of the suspected malfunction and ensures that the investigation and handling based on this are targeted.
[0096] Furthermore, before obtaining the first type of fault investigation result obtained by investigating and processing the suspected fault cause, the method further includes the following steps: First, generating work order information corresponding to the first type of suspected fault cause; then, sending the work order information to the operation and maintenance node, so that the operation and maintenance node can obtain the target spare part according to the work order information and use the target spare part to perform fault investigation and processing on the fan assembly.
[0097] When a first-type suspected fault exists, troubleshooting needs to be conducted based on the fan assembly, which is considered a suspected component, to obtain the first-type fault diagnosis results. For example, replace the fan assembly and then check if the temperature alarm no longer occurs for the target component. The work order information describes the need to replace the fan assembly or several fans within the fan assembly. The operations and maintenance node receives the work order information, then obtains the new fan assembly or several new fans as target spare parts, and replaces the original related components in the target server with the target spare parts.
[0098] In practical applications, the server-side fault handling system can generate and send work order information. See also... Figure 6 , 8 The system includes the following instructions: Hardware Replacement Work Order Creation Instruction: Once the specific component requiring replacement is identified, a corresponding component replacement work order will be created. Whole Machine Operation Authorization Instruction: Regardless of whether replacing the component involves downtime, operation authorization must be requested from the business team. On-site Processing Work Order & Spare Parts Outbound Work Order Instruction: After business authorization is granted, an on-site processing work order and a spare parts outbound work order will be generated. One grants operation authorization to the on-site engineer, and the other can be used to quickly request spare parts. Maintenance Engineer Replacement Instruction: On-site personnel in the data center will perform the replacement operation on the component. Completion and Acceptance Instruction: Different fault types will have different testing logic, mainly focusing on machine availability and component quantity. If successful, the process will end directly; otherwise, it will be returned for rework.
[0099] See Figure 9In the process of "Node 8" -> "Node 11" -> "Node 12", if the verification result indicates that the target component has a temperature alarm, and no temperature alarm logs indicating other components are received after a certain buffer time, it is determined that the number of components with temperature alarms is 1. In this case, it is necessary to investigate based on the target component, such as replacing the old target component with the new one and then checking whether the new target component has a temperature alarm. See also... Figure 9 In the process of "Node 8" -> "Node 10", when the verification result indicates that the target component has a temperature alarm, and after a certain buffer time, a temperature alarm log indicating other components is received, it is determined that the number of components with temperature alarms is greater than 1. The component replacement involved here can be referred to the above-mentioned "Work Order Information" content, which will not be repeated here.
[0100] In addition, see Figure 9 In the process of "Node 8" to "Node 10", when the first type of fault investigation result indicates that there is no first type of suspected fault cause, a software fault investigation instruction is generated; then, the software fault investigation instruction is sent to the R&D node so that the R&D node can perform software-level fault investigation processing on the specified server cluster based on the software fault investigation instruction.
[0101] After replacing the fan assembly, the target component still displayed a temperature alarm, indicating that the first type of suspected fault could be ruled out from the "reasons for temperature alarms in the target component". Based on the logic of troubleshooting at the hardware level, troubleshooting at the software level can be considered. The server generates and sends software fault investigation commands to trigger the development nodes to perform software-level fault investigation on the specified server cluster.
[0102] In practical applications, the entity executing steps S202-S204 can be a server hardware fault diagnosis system on the server side. (See also...) Figure 9The process is as follows: 1) Node 1: On one hand, it collects relevant SDRs of the target server in real time. The SDRs are mainly used to determine the inlet temperature and fan speed. On the other hand, it receives other SNMPTRAP logs and data center temperature information after a certain buffer time. 2) Node 2: After a certain buffer time, it is automatically transferred from Node 1. 3) Nodes 1, 11, and 13 can use the same set of processing logic to replace components by calling the server fault handling system. Node 7 mainly refers to fan replacement, Node 11 mainly refers to target component replacement, and Node 13 mainly refers to motherboard replacement. 4) Nodes 9 and 12 can use the same set of processing logic to trigger the collection of relevant SDRs of the target server. Based on the relevant SDRs, it determines whether the real-time temperature of the target component has returned to normal. If it has returned to normal, the process ends directly. If it has not returned to normal, Node 9 goes to Node 8, and Node 12 goes to Node 13. 5) Node 10: Since the influence of ambient temperature and heat dissipation has been eliminated at this time, if the number of components with temperature alarms is greater than 1, it should be considered a difficult problem. Then the online diagnostic process goes to Node 10. This process utilizes an online diagnostic system. The system's main function is to quickly reach server development with complex issues via work orders, improving diagnostic efficiency while enriching the development case library. This helps to avoid complex issues from the design stage or improve the efficiency of automated diagnosis. When server development indicates that a component needs replacement, the process is routed to node 11. If hardware replacement is not involved, but only firmware updates or other troubleshooting tests, the main process ends, and subsequent processes follow. The fault diagnosis solution provided in this application improves the efficiency of component temperature alarm diagnosis, reducing the overall average diagnosis time from 24 hours to approximately 1 hour, including the time spent on diagnosis and replacement. It also saves dedicated manpower while expanding the monitoring scope. It shifts from passive response to proactive monitoring, reducing business complaints. Furthermore, the fault diagnosis solution provided in this application can also generate temperature alarms through real-time collected SDR data.
[0103] As can be seen from the technical solutions provided in the above embodiments of this application, this application determines the target component and target server indicated by the temperature alarm log, and then obtains the first type of sensor information of the target component, the second type of sensor information of the suspicious component, and the temperature information of the deployment unit to which the target server belongs, thereby determining the fault investigation result for the temperature alarm log based on the obtained information. The target component is the server component in the target server that triggered the temperature alarm, and the suspicious component is the server component in the target server that has a suspected correlation with the temperature alarm that triggered the target component. This application provides a programmed fault investigation scheme, which can avoid the interference of personal subjective judgment, reduce the occurrence of missed and false judgments, and thus achieve accurate fault investigation. The reduction of manual intervention, the automation of the entire fault investigation process, and the ability to handle concurrent processing can improve fault investigation efficiency and save labor costs.
[0104] This application also provides a fault diagnosis device, such as... Figure 10 As shown, the fault diagnosis device 1000 includes:
[0105] First determining module 1001: In response to a temperature alarm log for a specified server cluster, determining the target component and target server indicated by the temperature alarm log; wherein, the target component is the server component in the target server that has a temperature alarm, and the target server is any server in the specified server cluster;
[0106] First acquisition module 1002: used to acquire first type of sensor information of the target component and second type of sensor information of the suspicious component; wherein, the suspicious component is a server component in the target server that has a suspected correlation with the temperature alarm triggered by the target component;
[0107] Second acquisition module 1003: used to acquire temperature information of the deployment unit to which the target server belongs;
[0108] The second determining module 1004 is used to determine the fault investigation results for the temperature alarm log based on the first type of sensor information, the second type of sensor information, and the temperature information.
[0109] It should be noted that the apparatus and method embodiments described in the device embodiments are based on the same inventive concept.
[0110] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the troubleshooting method provided in the above method embodiments.
[0111] Furthermore, Figure 11 A schematic diagram of the hardware structure of an electronic device for implementing the fault diagnosis method provided in the embodiments of this application is shown. The electronic device may participate in or include the fault diagnosis apparatus provided in the embodiments of this application. Figure 11 As shown, the electronic device 110 may include one or more processors 1102 (shown as 1102a, 1102b, ..., 1102n in the figure) (processor 1102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1104 for storing data, and a transmission device 1106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 11 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 110 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.
[0112] It should be noted that the aforementioned one or more processors 1102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be wholly or partially integrated into any other element within the electronic device 110 (or mobile device). As involved in the embodiments of this application, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0113] The memory 1104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fault diagnosis method described in the embodiments of this application. The processor 1102 executes various functional applications and data processing by running the software programs and modules stored in the memory 114, thereby realizing the aforementioned fault diagnosis method. The memory 1104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1104 may further include memory remotely located relative to the processor 1102, and these remote memories can be connected to the electronic device 110 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0114] The transmission device 1106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 110. In one example, the transmission device 1106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 1106 may be a radio frequency (RF) module for wireless communication with the Internet.
[0115] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows a user to interact with the user interface of the electronic device 110 (or mobile device).
[0116] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a fault diagnosis method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the fault diagnosis method provided in the above method embodiment.
[0117] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0118] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
[0119] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and electronic device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0120] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0121] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A troubleshooting method, characterized in that, The method includes: In response to a temperature alarm log reported by the baseboard management controller of the target server, the out-of-band management system is used to determine the target component in the target server that has a temperature alarm; the target server is any server in the specified server cluster. The server hardware fault diagnosis system uses a first instruction to obtain the first type of sensor information of the target component and the second type of sensor information of the suspected component at the instruction issuance time, and uses a second instruction to obtain the temperature information of the rack or data center to which the target server belongs; the instruction issuance time is determined based on the time of receiving the temperature alarm log and a preset time interval, and the suspected component is a server component in the target server that is suspected of being related to the temperature alarm that caused the target component to occur; When the temperature value indicated by the first type of sensor information is higher than the first preset quantity threshold, the server hardware fault diagnosis system uses the comparison results of the second type of sensor information and the second preset quantity threshold, as well as the comparison results of the temperature information and the third preset quantity threshold, to determine whether there is a suspicious fault cause that causes the target component to trigger a temperature alarm. When a suspected fault is found, the server hardware fault diagnosis system is used to obtain a first type of fault investigation result obtained by investigating and processing the suspected fault. The first type of fault investigation result indicates whether the target component has a temperature alarm after the suspected component is replaced, and / or whether the target component has a temperature alarm after the ambient temperature is adjusted to the normal range. When no suspicious cause of failure is found, the server hardware fault diagnosis system is used to obtain the second type of fault diagnosis result obtained by investigating and processing the target component. The second type of fault diagnosis result indicates whether the new target component will have a temperature alarm after the target component is replaced.
2. The method according to claim 1, characterized in that, The method further includes: The server hardware fault diagnosis system sends the first instruction to the sensor information management module; the sensor information management module is used to manage the sensor information obtained by monitoring the server components in the target server. The server hardware fault diagnosis system sends the second instruction to the temperature management system corresponding to the rack or computer room to which the target server belongs.
3. The method according to any one of claims 1 or 2, characterized in that: Before determining the target component in the target server that triggered the temperature alarm using the out-of-band management system in response to a received temperature alarm log reported by the baseboard management controller of the target server, the method further includes: The temperature alarm log is received using the out-of-band management system; the temperature alarm log is a log based on the Simple Network Management Protocol (SMMP) trap format. The step of responding to a received temperature alarm log reported by the baseboard management controller of the target server, and using an out-of-band management system to determine the target component in the target server where the temperature alarm occurred, includes: The out-of-band management system is used to parse the temperature alarm log to obtain a first field indicating the server identifier and a second field indicating the component temperature alarm. The out-of-band management system determines the target server based on the first field; The out-of-band management system is used to match the second field with a set of reference fields indicating component temperature alarms, and when there is a reference field in the set of reference fields that matches the second field, the target component is determined based on the component indicated by the reference field.
4. The method according to claim 1, characterized in that, The method further includes: When the temperature value indicated by the first type of sensor information is lower than or equal to the first preset quantity threshold, the server hardware fault diagnosis system is used to update the number of false alarms of the target server. The server hardware fault diagnosis system is used to determine the monitoring time corresponding to the first type of sensor information, and the monitoring period to which the monitoring time belongs; When the number of false alarms to the target server within the monitoring period exceeds the fourth preset threshold, the server hardware fault diagnosis system is used to trigger the steps of comparing the second type of sensor information with the second preset threshold and comparing the temperature information with the third preset threshold.
5. The method according to claim 1, characterized in that, The suspected components include an air inlet assembly and a fan assembly. The second type of sensor information includes the temperature value of the air inlet assembly and the rotation speed value of the fan assembly. The second preset quantity threshold includes the temperature threshold of the air inlet assembly and the rotation speed threshold of the fan assembly. The method of using the server hardware fault diagnosis system to determine whether there is a suspicious fault cause that could trigger a temperature alarm in the target component, based on the comparison results of the second type of sensor information and the second preset quantity threshold, and the comparison results of the temperature information and the third preset quantity threshold, includes: When the temperature value is less than or equal to the temperature threshold, the server hardware fault diagnosis system is used to determine whether the rotation speed value is less than the rotation speed threshold; if it is less, the server hardware fault diagnosis system is used to identify the fan assembly failure as the first type of suspected fault cause that triggers the temperature alarm of the target component. When the temperature value is greater than the temperature threshold, the server hardware fault diagnosis system uses the comparison result between the temperature information and the third preset quantity threshold to determine whether the ambient temperature is normal; if it is normal, the server hardware fault diagnosis system uses the speed value to determine whether it is less than the speed threshold; if it is less, the server hardware fault diagnosis system uses the fan component failure as the first type of suspected fault cause that triggers the temperature alarm of the target component. When the temperature value is greater than the temperature threshold, the server hardware fault diagnosis system uses the comparison result between the temperature information and the third preset quantity threshold to determine whether the ambient temperature is normal; if it is not normal, the server hardware fault diagnosis system uses the abnormal ambient temperature as a second type of suspected fault cause that triggers the temperature alarm of the target component.
6. The method according to claim 5, characterized in that, Before obtaining the first type of fault diagnosis result obtained by investigating and processing the suspected fault cause using the server hardware fault diagnosis system, the method further includes: The server hardware fault diagnosis system is used to generate work order information corresponding to the first type of suspected fault causes. The server hardware fault diagnosis system sends the work order information to the operation and maintenance node, so that the operation and maintenance node can obtain the target spare part based on the work order information and use the target spare part to troubleshoot the fan assembly.
7. The method according to claim 6, characterized in that, The method further includes: When the first type of fault investigation result indicates that there is no first type of suspected fault cause, the server hardware fault diagnosis system is used to generate a software fault investigation instruction. The server hardware fault diagnosis system is used to send the software fault diagnosis command to the development node, so that the development node can perform software-level fault diagnosis and processing on the specified server cluster based on the software fault diagnosis command.
8. A fault diagnosis device, characterized in that, The device includes: First determination module: In response to a temperature alarm log reported by the baseboard management controller of the target server, the module uses an out-of-band management system to determine the target component in the target server that has triggered a temperature alarm; the target server is any server in the specified server cluster. Acquisition Module: Used by the server hardware fault diagnosis system to acquire first-type sensor information of the target component and second-type sensor information of the suspected component by sending a first instruction at the instruction issuance time, and to acquire temperature information of the rack or data center to which the target server belongs by sending a second instruction; the instruction issuance time is determined based on the time of receiving the temperature alarm log and a preset time interval, and the suspected component is a server component in the target server that is suspected of being related to the temperature alarm that caused the target component to occur; The second determining module is used to determine, when the temperature value indicated by the first type of sensor information is higher than a first preset quantity threshold, whether there is a suspected fault cause that could trigger a temperature alarm in the target component, based on the comparison results of the second type of sensor information and the second preset quantity threshold, and the comparison results of the temperature information and the third preset quantity threshold, using the server hardware fault diagnosis system; when a suspected fault cause exists, the server hardware fault diagnosis system is used to obtain a first type of fault investigation result obtained by investigating the suspected fault cause, the first type of fault investigation result indicating whether the target component triggers a temperature alarm after replacing the suspected component, and / or whether the target component triggers a temperature alarm after adjusting the ambient temperature to a normal range; when no suspected fault cause exists, the server hardware fault diagnosis system is used to obtain a second type of fault investigation result obtained by investigating the target component, the second type of fault investigation result indicating whether the new target component triggers a temperature alarm after the target component is replaced.
9. The apparatus according to claim 8, characterized in that, The device is also used for: The server hardware fault diagnosis system sends the first instruction to the sensor information management module; the sensor information management module is used to manage the sensor information obtained by monitoring the server components in the target server. The server hardware fault diagnosis system sends the second instruction to the temperature management system corresponding to the rack or computer room to which the target server belongs.
10. The apparatus according to any one of claims 8 or 9, characterized in that: The apparatus is further configured to: before determining the target component in the target server that has a temperature alarm using the out-of-band management system in response to a received temperature alarm log reported by the baseboard management controller of the target server, the apparatus further includes: receiving the temperature alarm log using the out-of-band management system; the temperature alarm log is a log based on the Simple Network Management Protocol Trap Format; The first determining module is configured to: use the out-of-band management system to parse the temperature alarm log to obtain a first field indicating server identification and a second field indicating component temperature alarm; use the out-of-band management system to determine the target server based on the first field; use the out-of-band management system to match the second field with a set of reference fields indicating component temperature alarm, and when there is a reference field in the set of reference fields that matches the second field, determine the target component based on the component indicated by the reference field.
11. The apparatus according to claim 8, characterized in that, The device is also used for: When the temperature value indicated by the first type of sensor information is lower than or equal to the first preset quantity threshold, the server hardware fault diagnosis system is used to update the number of false alarms of the target server. The server hardware fault diagnosis system is used to determine the monitoring time corresponding to the first type of sensor information, and the monitoring period to which the monitoring time belongs; When the number of false alarms to the target server within the monitoring period exceeds the fourth preset threshold, the server hardware fault diagnosis system is used to trigger the steps of comparing the second type of sensor information with the second preset threshold and comparing the temperature information with the third preset threshold.
12. The apparatus according to claim 8, characterized in that, The suspected components include an air inlet assembly and a fan assembly. The second type of sensor information includes the temperature value of the air inlet assembly and the rotation speed value of the fan assembly. The second preset quantity threshold includes the temperature threshold of the air inlet assembly and the rotation speed threshold of the fan assembly. The method of using the server hardware fault diagnosis system to determine whether there is a suspicious fault cause that could trigger a temperature alarm in the target component, based on the comparison results of the second type of sensor information and the second preset quantity threshold, and the comparison results of the temperature information and the third preset quantity threshold, includes: When the temperature value is less than or equal to the temperature threshold, the server hardware fault diagnosis system is used to determine whether the rotation speed value is less than the rotation speed threshold; if it is less, the server hardware fault diagnosis system is used to identify the fan assembly failure as the first type of suspected fault cause that triggers the temperature alarm of the target component. When the temperature value is greater than the temperature threshold, the server hardware fault diagnosis system uses the comparison result between the temperature information and the third preset quantity threshold to determine whether the ambient temperature is normal; if it is normal, the server hardware fault diagnosis system uses the judgment of whether the rotation speed value is less than the rotation speed threshold; if it is less, the fan assembly failure is regarded as the first type of suspected fault cause that triggers the temperature alarm of the target component. When the temperature value is greater than the temperature threshold, the server hardware fault diagnosis system uses the comparison result between the temperature information and the third preset quantity threshold to determine whether the ambient temperature is normal; if it is not normal, the server hardware fault diagnosis system uses the abnormal ambient temperature as a second type of suspected fault cause that triggers the temperature alarm of the target component.
13. The apparatus according to claim 12, characterized in that, The device is also used for: Before obtaining the first type of fault investigation result by using the server hardware fault diagnosis system to investigate and process the suspected fault cause, the server hardware fault diagnosis system generates work order information corresponding to the first type of suspected fault cause. The server hardware fault diagnosis system sends the work order information to the operation and maintenance node, so that the operation and maintenance node can obtain the target spare part based on the work order information and use the target spare part to troubleshoot the fan assembly.
14. The apparatus according to claim 13, characterized in that, The device is also used for: When the first type of fault investigation result indicates that there is no first type of suspected fault cause, the server hardware fault diagnosis system is used to generate a software fault investigation instruction. The server hardware fault diagnosis system is used to send the software fault diagnosis command to the development node, so that the development node can perform software-level fault diagnosis and processing on the specified server cluster based on the software fault diagnosis command.
15. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the troubleshooting method as described in any one of claims 1-7.
16. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the troubleshooting method as described in any one of claims 1-7.
17. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads and executes the computer instructions from the computer-readable storage medium, causing the computer device to perform the troubleshooting method as described in any one of claims 1-7.
Citation Information
Patent Citations
Method for generating failure log of server and related equipment
CN108897666A
Server cluster monitoring and management system
CN111984498A
Fault root cause determination method and device, server and computer readable medium
CN112448836A