A fault processing method, apparatus, device and medium

Abstract

The application discloses a fault processing method, device, equipment and medium, and belongs to the technical field of servers. The method comprises the following steps: setting a security network configuration item and an error processing configuration item for a manager port; in response to receiving fault alarm information through the manager, obtaining processor register information, wherein the processor register information comprises a current fault diagnosis state, and determining whether to collect fault diagnosis source data according to the current fault diagnosis state; in response to collecting the fault diagnosis source data, determining whether to perform a soft restart on the system according to the error processing configuration item; in response to performing the soft restart on the system according to the security network configuration item, determining a fault type according to the fault diagnosis source data, and automatically analyzing and processing the fault according to the fault type. The application can realize compatibility of hardware and software, and can automatically collect fault source data and process the fault after the fault occurs, thereby improving the correctness of fault processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of servers, and in particular to a fault processing method and device, equipment and a medium. BACKGROUND

[0002] At present, there are various storage devices and computing devices on servers, which are provided and developed by different manufacturers and different research and development teams. However, unpredictable errors may occur during CPU operation, which may cause the server to fail to operate normally.

[0003] In related technologies, fault register information is captured during the BIOS system restart process, and then the information is transmitted to the baseboard management controller through the IPMI channel to record logs. However, the register information needs to be trimmed at the BIOS end, and needs to fully comply with the IPMI protocol for information transmission, so that the capacity and integrity of the log information are limited.

[0004] At the same time, the system has been loaded, and the processor needs to be soft restarted to reload the operating system to obtain error information in the register, and then the register information is transmitted to the manager to record logs and issue alarms and provide diagnostic suggestions. Therefore, there is no way to preserve the original error environment and no way to analyze and accurately locate the fault cause in a timely manner when the problem occurs. SUMMARY

[0005] In order to overcome the above technical defects, the purpose of the present application is to provide a fault processing method, device, equipment and medium. The method comprises: setting a security network configuration item and an error processing configuration item on a manager port, wherein the security network configuration item is used to prevent changes in the down-restart environment, and the error processing configuration item is used to determine different restart strategies according to error types; in response to receiving fault alarm information through the manager, processor register information is obtained, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery and fault isolation; whether to collect fault diagnosis source data is determined according to the current fault diagnosis state; in response to collecting fault diagnosis source data, whether to soft restart the system is determined according to the error processing configuration item; in response to soft restarting the system according to the security network configuration item, a fault type is determined according to the fault diagnosis source data, and the fault is automatically analyzed and processed according to the fault type. The present application can realize automatic collection of fault source data by the manager after the occurrence of a fault, and improve the correctness of fault processing through the compatibility of hardware and software.

[0006] The specific technical solutions provided by the embodiments of the present application are as follows:

[0007] In a first aspect, the application provides a fault processing method, the method comprising:

[0008] setting a security network configuration item and an error processing configuration item for the manager port, wherein the security network configuration item is used to prevent changes in the down-restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0009] in response to receiving fault alarm information through the manager, obtaining processor register information, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery, and fault isolation;

[0010] determining whether to collect fault diagnosis source data according to the current fault diagnosis state;

[0011] in response to collecting fault diagnosis source data, determining whether to perform a soft restart of the system according to the security network configuration item according to the error processing configuration item;

[0012] in response to performing a soft restart of the system according to the security network configuration item, determining a fault type according to the fault diagnosis source data, and performing automatic analysis and processing of the fault according to the fault type.

[0013] In one embodiment, before setting the security network configuration item and the error processing configuration item for the manager port, the method comprises:

[0014] setting a fault diagnosis function in the manager, recording state information at the time of a fault, and monitoring the fault alarm information; importing a crash dump error code in the manager, and monitoring the fault diagnosis state;

[0015] enabling the security network configuration item, and when a number of flooding attack requests occur in the system, triggering a processor reset and clearing flooding attack resources;

[0016] enabling the error processing configuration item, and when a number of fault errors occur in the system, protecting the system by resetting a PCIe controller.

[0017] In one embodiment, the fault diagnosis source data includes a plurality of fault data blocks, the fault data blocks include state data and Internet Protocol identifier data, and the method of determining a fault type according to the fault diagnosis source data and performing automatic analysis and processing of the fault according to the fault type comprises:

[0018] judging the validity of the fault data blocks according to the state data;

[0019] In response to the valid fault data block, a fault type is determined according to the state data, the Internet protocol identifier data and the standard document, and the fault is analyzed according to the fault type;

[0020] The analyzing the fault according to the fault type comprises:

[0021] Analyzing a fault memory location according to the fault type;

[0022] Analyzing a fault severity level and a fault code according to the fault type;

[0023] Determining a fault detailed description according to the fault type;

[0024] Recording a fault log according to the fault memory location, the fault severity level, the fault code and the fault detailed description;

[0025] Formulating a fault processing solution according to the fault log.

[0026] In one embodiment, the method comprises:

[0027] In response to no soft restart of the system, a down interface is reserved, down processor register information is collected by a collection tool remotely or on site, the down processor register information is analyzed, and the fault is processed.

[0028] In one embodiment, the determining the fault type according to the fault diagnosis source data and automatically analyzing the fault according to the fault type further comprises:

[0029] The fault diagnosis source data is cleaned and formatted to remove data noise, errors and redundant information;

[0030] Fault features are extracted from the cleaned fault diagnosis source data, wherein the fault features comprise trend analysis, statistical indicators and frequency domain analysis of time series data;

[0031] The fault features are classified by a fault mode training model to determine the fault type;

[0032] A fault code manual or knowledge base is obtained according to the fault type, and a fault log file is analyzed according to the fault code manual or knowledge base;

[0033] The fault is analyzed according to a comparison between the fault log file and historical fault cases;

[0034] Whether the fault processing is completed is verified by retesting;

[0035] The checking whether the fault processing is completed by retesting includes:

[0036] Reproducing the condition when the fault occurs, and confirming whether the same fault problem occurs;

[0037] Obtaining the log file of the application program, and confirming whether the abnormal record occurs;

[0038] Monitoring the CPU usage, the memory usage, the disk and the network traffic of the system by the performance monitoring tool, and confirming whether the system performance returns to normal;

[0039] Performing the stress test on the system, and confirming whether the system can run;

[0040] Confirming whether all the system configuration files are modified and saved.

[0041] In one embodiment, after determining the fault type according to the fault diagnosis source data and automatically analyzing and processing the fault according to the fault type, the method further includes:

[0042] Recording the fault error log by the manager, and optimizing the fault processing according to the fault error log;

[0043] The optimizing the fault processing according to the fault error log includes:

[0044] Obtaining the fault error log;

[0045] Structuring the fault error log by unifying the log format of the fault error log;

[0046] Performing the aggregation and statistical analysis on the structured fault error log data, and determining the fault mode and rule;

[0047] Creating the fault tree according to the fault mode and rule to analyze the fault, and optimizing the fault processing strategy.

[0048] In one embodiment, the method includes:

[0049] When the automatic processing of the fault has a problem, performing the soft restart on the system, obtaining the processor register information, determining the fault error type and the fault component according to the processor register information;

[0050] Creating the first fault information according to the fault error type and the fault component information;

[0051] Sending the first fault information to the manager;

[0052] The manager records the fault log according to the first fault information, and analyzes the fault log;

[0053] and replacing the faulty component according to the faulty component information.

[0054] In a second aspect, the present application provides a fault processing device, comprising:

[0055] a setting module, configured to set a secure network configuration item and an error processing configuration item for a manager port, wherein the secure network configuration item is used to prevent changes in a down-restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0056] a receiving module, configured to acquire processor register information in response to receiving fault alarm information through the manager, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery and fault isolation;

[0057] a judging module, configured to judge whether to collect fault diagnosis source data according to the current fault diagnosis state;

[0058] in response to collecting the fault diagnosis source data, judging whether to perform a soft restart according to the secure network configuration item;

[0059] a processing module, configured to determine a fault type according to the fault diagnosis source data in response to performing a soft restart according to the secure network configuration item, and perform automatic analysis and processing on the fault according to the fault type.

[0060] In a third aspect, the present application provides a fault processing device, comprising:

[0061] one or more processors;

[0062] a storage device, configured to store one or more programs;

[0063] when the one or more programs are executed by the one or more processors, the one or more processors implement the fault processing method according to any one of the first aspect.

[0064] In a fourth aspect, the present application provides a computer device, comprising:

[0065] a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the fault processing method according to any one of the first aspect.

[0066] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the fault processing method according to any one of the first aspect.

[0067] In a sixth aspect, the present application provides a computer storage medium, which comprises:

[0068] A computer program is stored on the computer storage medium, and the computer program is executed by a processor to implement the steps of the fault processing method according to any one of the first aspect.

[0069] Compared with the prior art, the method provided by the embodiment of the present application comprises: setting a security network configuration item and an error processing configuration item for a manager port, wherein the security network configuration item is used to prevent changes in a downtime restart environment, and the error processing configuration item is used to determine different restart strategies according to error types; in response to receiving fault alarm information through the manager, processor register information is acquired, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery and fault isolation; whether to collect fault diagnosis source data is determined according to the current fault diagnosis state; in response to collecting the fault diagnosis source data, whether to soft restart the system is determined according to the error processing configuration item; in response to soft restarting the system according to the security network configuration item, a fault type is determined according to the fault diagnosis source data, and the fault is automatically analyzed and processed according to the fault type. Through joint debugging and compatibility on hardware and software, the present application can realize that, after a fault occurs, a baseboard management controller can automatically collect fault source data, integrate analysis code in the manager, record detailed logs after automatically analyzing fault data, assist in fault diagnosis and fault solving, and avoid the problem of low support efficiency of technicians who go to a customer site after a fault occurs.

[0070] The technical scheme provided by the embodiment of the present application can more accurately provide guidance for on-site operation and maintenance by detecting a processor running state through a baseboard management controller, automatically collecting and analyzing processor register information when a server is down, recording detailed fault logs, error contents, error causes, maintaining logs and giving diagnosis suggestions. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0072] Figure 1 The first flowchart of the fault processing method provided by Embodiment One of the present application;

[0073] Figure 2A second flowchart of the fault processing method provided for Embodiment Two of the present application;

[0074] Figure 3 A third flowchart of the fault processing method provided for Embodiment Two of the present application;

[0075] Figure 4 A fourth flowchart of the fault processing method provided for Embodiment Two of the present application;

[0076] Figure 5 A structure diagram of the fault processing device provided for Embodiment Three of the present application;

[0077] Figure 6 An exemplary system that can be used to implement various embodiments described herein is provided for Embodiment Seven of the present application. DETAILED DESCRIPTION

[0078] For the purposes of the present application, the technical solutions and advantages thereof will be more clearly described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0079] It should be noted that, unless the context clearly requires otherwise, throughout the description and the claims of this specification, the words "comprise", "comprising", and the like should be construed to be inclusive, rather than exclusive or exhaustive. That is, the term "comprises" should be construed to mean "including but not limited to".

[0080] In addition, in the description of the present application, the meaning of "multiple" is two or more, unless otherwise specified.

[0081] Firstly, when a fault occurs, if the machine does not restart, the register information needs to be collected remotely or on-site by the R&D personnel, at which time the downtime environment needs to be preserved, and most problems occur in the customer's computer room, which greatly inconveniences the efficiency of problem solving for customer production and R&D.

[0082] Secondly, when a fault occurs, if the machine performs a soft restart, the fault register information is parsed by the operating system BIOS during the restart process, but during the system restart process, the re-execution of the processor CPU causes part of the fault register information to be refreshed and overwritten, which cannot completely reproduce the downtime environment at that time, affecting the R&D positioning of the root cause, at which time the problem needs to be reproduced by stress testing in the R&D laboratory, and most of the problems have a small probability and are not easy to reproduce in the laboratory environment of one or two machines, which affects the positioning and solving of the root cause of the problem.

[0083] Finally, the system BIOS parses the fault register information and transmits the fixed-format information to the BMC via the IPMB channel to record the log. Due to the limitations of IPMI, the SEL log information transmitted by the BIOS is relatively limited, and can only record the approximate faulty component and rough error information. Some errors do not require component replacement, but because the SEL log cannot record in detail, on-site maintenance can only replace the component based on the component location recorded in the log, so there is no more accurate and efficient solution.

[0084] Example 1

[0085] This application provides a fault handling method, such as... Figure 1 As shown, the method includes:

[0086] Configure security network settings and error handling settings for the manager port. The security network settings are used to prevent changes in the environment during a system crash restart, and the error handling settings are used to determine different restart strategies based on the error type.

[0087] In response to receiving fault alarm information through the manager, processor register information is obtained, wherein the processor register information includes the current fault diagnosis status, and the fault diagnosis status includes at least one of error detection, error reporting, error recovery and fault isolation;

[0088] Determine whether to collect fault diagnosis source data based on the current fault diagnosis status;

[0089] In response to the collection of fault diagnosis source data, the system is then judged whether to perform a soft restart based on the security network configuration item according to the error handling configuration item.

[0090] In response to a soft reboot of the system based on the security network configuration items, the fault type is determined based on the fault diagnosis source data, and the fault is automatically analyzed and processed based on the fault type.

[0091] Specifically, this application uses the Baseboard Management Controller (BMC) to detect the CPU's operating status. When a system crashes, it automatically collects and analyzes CPU register information. Based on the analysis results, it records detailed fault logs, error content, error causes, maintenance logs, and diagnostic suggestions, providing more accurate guidance for on-site operation and maintenance.

[0092] It is understandable that this application enables the BMC manager to automatically collect fault source data after a fault occurs through hardware and software integration and compatibility, fulfilling the necessary conditions for fault diagnosis on the BMC side, as well as the complete hardware and software operation process and fault source data parsing process after fault diagnosis occurs.

[0093] The baseboard management controller (BMC) plays a role of a housekeeper in the server, can manage the power supply, temperature and other performances of the whole system, and can play a role of a watchdog to restart the CPU when the system crashes, wherein the baseboard management controller is an independent system, does not depend on other hardware (such as the CPU, the memory and the like) on the system, and does not depend on the BIOS and the OS, but the baseboard management controller can interact with the BIOS and the OS, so that better platform management can be achieved, and system management software under the OS can cooperate with the BMC to achieve better management effect.

[0094] The technical scheme provided by the embodiment of the application has the beneficial effects that:

[0095] The application can realize automatic collection of fault source data by the baseboard management controller after a fault occurs, realize automatic recording of detailed logs after automatic analysis of the fault data by integrating analysis codes in the manager, and assist in fault diagnosis and fault solving, thereby avoiding the problem of low support efficiency of technicians after a fault occurs.

[0096] The technical scheme provided by the embodiment of the application can more accurately provide guidance for on-site operation and maintenance by detecting the running state of the processor by the baseboard management controller, automatically collecting and analyzing processor register information when the system crashes, recording detailed fault logs, error contents, error causes, maintaining logs and giving diagnosis suggestions.

[0097] Embodiment two

[0098] The embodiment of the application provides a fault processing method, as shown in Figure 2 The method comprises the following steps:

[0099] In step S01, a fault diagnosis function is set in the manager, state information when a fault occurs is recorded, and the fault alarm information is monitored; a crash dump error code is imported in the manager, and the fault diagnosis state is monitored.

[0100] The security network configuration item is started, when a plurality of flooding attack requests occur in the system, the processor is reset, and the flooding attack resource is cleared.

[0101] The error processing configuration item is started, when a plurality of fault errors occur in the system, the PCIe controller is reset, and the system is protected.

[0102] Specifically, a crashdump code is integrated in the baseboard management controller for monitoring an APML_ALERT (advanced platform management link) warning and collecting CPU register data; wherein the crashdump code generally refers to a piece of data or code generated by the system automatically when a software or hardware failure occurs, for recording state information at the time of failure.

[0103] A crashdump code is imported in the BMC, and a RAS (Reliability, Availability, and Serviceability) daemon is implemented for real-time monitoring of the APML_ALERT.

[0104] An enable APML_ALERT identifier Pin is implemented when a fatal error occurs in the CPU, which is provided by the CPU manufacturer and varies with different CPU manufacturers.

[0105] In step S02, a secure network configuration item and an error handling configuration item are set for the manager port, wherein the secure network configuration item is used to prevent changes in the crash restart environment, and the error handling configuration item is used to determine different restart strategies according to error types.

[0106] Specifically, a secure network configuration item PcdResetCpuOnSyncFlood is set at the BMC end to prevent the BMC from resetting the SOC (system on chip) during the collection of crash register information, which causes the management controller and core system chip in the server or computing platform to be reset or restarted, resulting in changes in the crash environment and affecting fault diagnosis; wherein the secure network configuration item is a configuration item or a function identifier related to the security or network equipment of a computer system, and the PcdResetCpuOnSyncFlood configuration item can be used to specify whether the system should take measures to reset the CPU state or perform other protective operations to prevent the system from crashing due to resource exhaustion, which usually involves the security policy setting of network equipment or servers, for improving the robustness and security of the system.

[0107] A BMC end setting error handling configuration item PcdAmdPcieSyncFloodOnFatal, different restart strategies after different errors, according to different customer needs, after the occurrence of the outage problem, the outage site is retained; wherein the error handling configuration item is an AMD PCIe (Peripheral Component Interconnect Express) related configuration item, which usually appears in the firmware or BIOS settings of AMD platform, and the configuration item is mainly used to handle the situation of Sync Flood Attack or Fatal Error on the PCIe link.

[0108] Step S03, receiving fault alarm information through the manager;

[0109] Obtaining processor register information through a platform management link, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery and fault isolation;

[0110] According to the current fault diagnosis state, it is judged whether to collect fault diagnosis source data.

[0111] Specifically, Fatal Error triggers a microcontroller SMU (System Management Unit) interrupt, after the SMU interrupt is triggered, the logic element CPLD receives the interrupt signal, pulls up the APML_ALERT alarm information, and the BMC monitors that the APML_ALERT alarm information is pulled up, and then obtains the fault diagnosis state Rasstatus (Reliability, Availability, and Serviceability, RAS features usually include error detection, error reporting, error recovery and fault isolation) through the advanced platform management link APML.

[0112] Step S04, in response to collecting fault diagnosis source data, then judging whether to soft restart the system according to the safety network configuration item according to the error handling configuration item;

[0113] In response to soft restarting the system according to the safety network configuration item, determining the fault type according to the fault diagnosis source data, and automatically analyzing and processing the fault according to the fault type.

[0114] Specifically, the BMC determines whether to collect fault source data according to a fault diagnosis state (Ras status) and whether to perform a soft restart or a hard restart; most faults require the collection of fault source data and then a soft restart, and after the collection of fault source data is completed, it is determined according to a user-defined setting whether to perform a restart;

[0115] When the user sets a restart after a downtime problem occurs, the BMC analyzes the collected data according to a fault diagnosis sel log analysis specification, analyzes the fault reason, the fault severity level, and points to the fault component according to different error types, and provides a diagnosis suggestion.

[0116] Step S04 further includes:

[0117] Step S041, judging the validity of the fault data block according to the state data;

[0118] In response to the fault data block being valid, determining a fault type according to the state data, an Internet protocol identifier data, and a standard document, and performing analysis processing on the fault according to the fault type;

[0119] The analysis processing on the fault according to the fault type includes:

[0120] Analyzing a fault memory location according to the fault type;

[0121] Analyzing a fault severity level and a fault code according to the fault type;

[0122] Determining a fault detailed description according to the fault type;

[0123] Recording a fault log according to the fault memory location, the fault severity level, the fault code, and the fault detailed description;

[0124] Formulating a fault processing solution according to the fault log.

[0125] Specifically, it is determined whether to collect fault diagnosis source data according to a fault state of a current processor CPU register, and after the collection of fault diagnosis data is completed, the fault data is analyzed; it is determined whether to restart according to a user setting, and after the machine is restarted after a fault occurs, the fault diagnosis source data and system BIOS transmission fault data are collected and processed, the BMC collected fault data is given priority, and the BIOS transmission data is processed after deduplication.

[0126] Here, the fault source data contains multiple fault data blocks, and the multiple data blocks are sequentially processed as follows: selecting key data from a single fault source data block for analysis (for example, each data block is a complete structure containing status, addr, ipid, synd, and other data), determining whether the current fault data block is valid fault data according to some bit bytes of the status data, if it is valid data, then determining the current fault type according to some bits of the status data and the ipid data in combination with the specification document provided by AMD; if the current fault data block is not valid data, the invalid data is discarded and does not need to be parsed;

[0127] After the fault type is determined, the fault pointing device, device location, error severity level, fault code, EXT fault type, and fault detailed description are determined according to the fault data and the parsing rule, and the above parsed data is recorded in a detailed log, and a fault processing suggestion is given.

[0128] For example: the memory Umc error needs to parse the fault memory location, such as which processor CPU, which channel, and which dimm; the fault severity level is parsed, such as CorrectError (CE), UnCorrectError (UCE), and FatalError; the fault code is parsed, such as 0C260600, corresponding to the location and severity level, etc., which is the log specification information; the EXT fault type is parsed, such as DramEcc, and the fault log is recorded according to the fault memory location, fault severity level, fault code, and fault detailed description; and a fault processing solution is developed according to the fault log.

[0129] Step S042, cleaning and formatting the fault diagnosis source data to remove data noise, errors, and redundant information;

[0130] Fault features are extracted from the cleaned fault diagnosis source data, wherein the fault features include trend analysis, statistical indicators, and frequency domain analysis of time series data;

[0131] The fault features are classified by a fault mode training model to determine the fault type;

[0132] A fault code manual or knowledge base is obtained according to the fault type, and a fault log file is analyzed according to the fault code manual or knowledge base;

[0133] The fault is analyzed and processed by comparing the fault log file with historical fault cases;

[0134] Whether the fault processing is completed is verified by retesting;

[0135] The checking whether the fault processing is completed by retesting includes:

[0136] Reproducing the condition when the fault occurs, and confirming whether the same fault problem occurs;

[0137] Obtaining the log file of the application program, and confirming whether the abnormal record occurs;

[0138] Monitoring the CPU usage, memory usage, disk and network traffic of the system through the performance monitoring tool, and confirming whether the system performance returns to normal;

[0139] Performing stress testing on the system, and confirming whether the system can run;

[0140] Confirming whether all system configuration files have been modified and saved.

[0141] Step S05, in response to not performing soft restart on the system, the down interface is retained, the down processor register information is collected through the collection tool remotely or on site, the down processor register information is analyzed, and the fault is processed.

[0142] Specifically, when the user sets not to restart after the down problem occurs, the BMC collects the fault data and does not process it, but retains the original fault data for the on-site operation and maintenance of the developer.

[0143] That is, the user sets not to restart after the down, the machine stops at the down interface after the machine is down, the on-site operation and maintenance personnel collects the down register information remotely or on site, and then analyzes the down register information to provide professional fault diagnosis opinions and suggestions to solve the down problem.

[0144] Step S06, recording the fault error log through the manager, and optimizing the fault processing according to the fault error log;

[0145] The optimization of the fault processing according to the fault error log includes:

[0146] Obtaining the fault error log;

[0147] Structuring the fault error log by unifying the log format;

[0148] Aggregating and statistically analyzing the structured fault error log data to determine the fault mode and trend rule;

[0149] Creating a fault tree according to the fault mode and trend rule to analyze the fault, and optimizing the fault processing strategy.

[0150] Step S07, when the automatic processing of the fault fails, then the system is soft restarted, the processor register information is obtained, the fault error type and the fault component are determined according to the processor register information;

[0151] The first fault information is created according to the fault error type and the fault component information;

[0152] The first fault information is sent to the manager;

[0153] The manager records the fault log according to the first fault information, and analyzes the fault log;

[0154] And the fault component is replaced according to the fault component information.

[0155] Specifically, the user sets the restart after the machine downtime, the machine will be soft restarted after downtime, the read register information is judged according to the fault analysis specification provided by the CPU manufacturer in the BIOS end, the approximate error type and the fault component are determined, then the information is combined, the combined information is sent to the manager BMC through the IPMB channel, the manager BMC records the system event log (SEL), and the on-site operation and maintenance personnel replace the fault component according to the prompt in the SEL log.

[0156] Wherein, the soft restart is a method of restarting the system through the operating system command, without closing the power of the server, the main steps of the soft restart include that the administrator or the user executes the restart command, such as reboot or shutdown-r now, or uses the system management tool to execute the soft restart operation; in the soft restart process, the operating system starts to close the currently running process, releases the resource, unloads the file system, etc., and then restarts the system; the main difference between the soft restart and the hard restart is that the hard restart is realized by forcibly closing the server and then restarting through the terminal, and the operating system and the application program are reloaded; while the soft restart is realized through the command of the operating system, and the system kernel and the hardware resource are not reinitialized, in the soft restart process, the BIOS is reloaded, the CPU register information is read during the loading process, the soft restart ensures that the CPU register information is reserved, and is used for fault analysis.

[0157] As Figure 3 , 4As shown, the technical scheme of the present application provides an automatic fault diagnosis method after a downtime problem occurs, and the related technical scheme is that after the downtime problem occurs, the BMC cannot collect fault data, and can only accept the BIOS transmission information during the startup process, and parse the log according to the BIOS transmission information, due to the limitation of the IPMI specification, only the approximate direction of the error reason in the log can be obtained, and the problem reason and the problem component cannot be confirmed in detail; in fact, after the downtime problem occurs, the log recorded by the BIOS cannot assist in analysis in many cases, and only the developers can carry professional equipment to read data on site, and then analyze the readable files by using professional tools to give diagnosis suggestions on site; and the current scheme greatly simplifies the fault diagnosis and analysis process, and through the joint debugging and compatibility of hardware and software, the BMC can automatically collect fault source data after the fault occurs, the analysis code is integrated in the BMC, detailed logs are recorded after the automatic analysis of the fault data, fault diagnosis and fault solving are assisted, and the problem of low fault handling efficiency caused by the fact that the technical personnel need to travel to the customer site for support after the fault occurs is avoided.

[0158] In addition, the manager port sets the security network configuration item and the error handling configuration item, including:

[0159] Logging into a baseboard management controller management interface;

[0160] Determining a setting item related to network security or attack protection in the manager interface;

[0161] Positioning to the security network configuration item and the error handling configuration item;

[0162] Configuring the security network configuration item (PcdResetCpuOnSyncFlood) and the error handling configuration item (PcdAmdPcieSyncFloodOnFatal);

[0163] The security network configuration item includes:

[0164] Setting the enable or disable function of the security network configuration item;

[0165] Setting a SYN flood attack threshold triggering a processor reset;

[0166] The error handling configuration item includes:

[0167] Setting the enable or disable function of the error handling configuration item;

[0168] Setting a threshold triggering a protection measure;

[0169] After the configuration is completed, the configuration information is saved, and whether the configuration item is set is verified.

[0170] Through joint debugging and compatibility on hardware and software, after a fault occurs, the baseboard management controller can automatically collect fault source data, integrate analysis code in the manager, record detailed logs after automatically analyzing the fault data, assist in fault diagnosis and fault resolution, and avoid the problem of low efficiency of technical personnel traveling to the customer site for support after a fault occurs.

[0171] The fault processing method provided by the embodiments of the present application can also be improved and optimized without departing from the technical solutions of the present application, and these improvements and optimizations should also be considered as the protection scope of the present application.

[0172] The technical solutions can also be applied to other terminal devices, and can be used for fault diagnosis of CPUs of other manufacturers and other PCIE devices.

[0173] The technical solutions provided by the embodiments of the present application have the following beneficial effects:

[0174] The present application can realize automatic collection of fault source data by the baseboard management controller after a fault occurs, integration of analysis code in the manager, recording of detailed logs after automatic analysis of fault data, assistance in fault diagnosis and fault resolution, and avoidance of the problem of low efficiency of technical personnel traveling to the customer site for support after a fault occurs.

[0175] The technical solutions provided by the embodiments of the present application can more accurately provide guidance for on-site operation and maintenance by detecting the running state of the processor by the baseboard management controller, automatically collecting processor register information and performing analysis when the processor is down, recording detailed fault logs, error content, error reason, maintaining logs, and giving diagnostic suggestions.

[0176] Embodiment three

[0177] The present application provides a fault processing device, as shown in Figure 5 The device comprises a setting module, a receiving module, a judging module, a processing module, and an optimization module.

[0178] In the present embodiment, the setting module is configured to set a security network configuration item and an error processing configuration item for the manager port, wherein the security network configuration item is used to prevent changes in the down-restart environment, and the error processing configuration item is used to determine different restart strategies according to error types.

[0179] The receiving module is configured to acquire processor register information in response to receiving fault alarm information through the manager, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery, and fault isolation.

[0180] a judging module, configured to judge whether to collect fault diagnosis source data according to the current fault diagnosis state;

[0181] in response to collecting the fault diagnosis source data, judging whether to perform soft restart according to the safety network configuration item;

[0182] a processing module, configured to, in response to performing soft restart according to the safety network configuration item, determine a fault type according to the fault diagnosis source data, and perform automatic analysis processing on the fault according to the fault type.

[0183] In one of the embodiments, a setting module is configured to set a fault diagnosis function in the manager, record state information when a fault occurs, and monitor the fault alarm information; and a crash dump error code is imported into the manager to monitor the fault diagnosis state.

[0184] the safety network configuration item is enabled, and when a plurality of flooding attack requests occur in the system, a processor is triggered to reset and clear the flooding attack resources;

[0185] the error processing configuration item is enabled, and when a plurality of fault errors occur in the system, the system is protected by resetting a PCIe controller.

[0186] In one of the embodiments, the processing module is configured to judge the validity of the fault data block according to the state data;

[0187] in response to the fault data block being valid, determining a fault type according to the state data, an internet protocol identifier data and a standard document, and performing analysis processing on the fault according to the fault type;

[0188] the analysis processing on the fault according to the fault type comprises:

[0189] analyzing a fault memory location according to the fault type;

[0190] analyzing a fault severity level and a fault code according to the fault type;

[0191] determining a fault detailed description according to the fault type;

[0192] recording a fault log according to the fault memory location, the fault severity level, the fault code and the fault detailed description;

[0193] formulating a fault processing solution according to the fault log.

[0194] In one of the embodiments, the processing module is configured to, in response to a soft restart not being performed on the system, retain a down interface, collect down processor register information through a collection tool remotely or on site, analyze the down processor register information, and process the fault.

[0195] In one of the embodiments, the processing module is configured to clean and format the fault diagnosis source data, remove data noise, errors, and redundant information.

[0196] The cleaned fault diagnosis source data is subjected to fault feature extraction, wherein the fault features include trend analysis, statistical indicators, and frequency domain analysis of time series data.

[0197] The fault features are classified through a fault mode training model to determine a fault type.

[0198] A fault code manual or knowledge base is obtained according to the fault type, and the fault log file is analyzed according to the fault code manual or knowledge base.

[0199] The fault is analyzed and processed according to a comparison between the fault log file and historical fault cases.

[0200] Whether the fault processing is completed is verified through retesting.

[0201] The verification of whether the fault processing is completed through retesting includes:

[0202] Conditions at the time of the fault occurrence are reproduced to confirm whether the same fault problem occurs.

[0203] A log file of an application program is obtained to confirm whether abnormal records occur.

[0204] The system performance is monitored through a performance monitoring tool to confirm whether the system performance returns to normal.

[0205] The system is subjected to stress testing to confirm whether the system can run.

[0206] It is confirmed whether all system configuration files have been modified and saved.

[0207] In one of the embodiments, the optimization module is configured to record fault error logs through a manager, and optimize the fault processing according to the fault error logs.

[0208] The optimization of the fault processing according to the fault error logs includes:

[0209] The fault error logs are obtained.

[0210] Structuring the fault error log unified log format;

[0211] Aggregating and statistically analyzing the structured fault error log data to determine fault modes and rules;

[0212] Creating a fault tree according to the fault modes and rules to perform fault analysis on the fault and optimize the fault handling strategy.

[0213] In one of the embodiments, the processing module is configured to perform a soft restart on the system, obtain processor register information, determine a fault error type and a fault component according to the processor register information, when there is a problem in the automatic processing of the fault;

[0214] Create first fault information according to the fault error type and the fault component information;

[0215] Send the first fault information to the manager;

[0216] The manager records a fault log according to the first fault information, and analyzes the fault log;

[0217] And replace the fault component according to the fault component information.

[0218] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0219] The present application can realize automatic collection of fault source data by the baseboard management controller after the fault occurs, integrate analysis code in the manager, realize recording of detailed logs after automatic analysis of fault data, assist in fault diagnosis and fault solving, and avoid the problem of low efficiency of technical personnel traveling to the customer site for support after the fault occurs.

[0220] The technical scheme provided by the embodiments of the present application can more accurately provide guidance for on-site operation and maintenance by detecting the processor running state through the baseboard management controller, automatically collecting and analyzing the processor register information when the system is down, recording detailed fault logs, error content, error reason, maintaining logs, and giving diagnostic suggestions.

[0221] Embodiment four

[0222] The present application also provides a fault processing device, comprising:

[0223] One or more processors;

[0224] A storage device for storing one or more programs;

[0225] When the one or more programs are executed by the one or more processors, the one or more processors perform the following fault processing method:

[0226] The manager port sets a security network configuration item and an error processing configuration item, wherein the security network configuration item is used to prevent changes of the down restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0227] In response to receiving fault alarm information through the manager, processor register information is acquired, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery and fault isolation;

[0228] According to the current fault diagnosis state, it is determined whether to collect fault diagnosis source data;

[0229] In response to collecting fault diagnosis source data, it is determined whether to perform soft restart of the system according to the security network configuration item according to the error processing configuration item;

[0230] In response to performing soft restart of the system according to the security network configuration item, a fault type is determined according to the fault diagnosis source data, and the fault is automatically analyzed and processed according to the fault type.

[0231] The technical scheme provided by the embodiment of the application has the following beneficial effects:

[0232] The application can realize automatic collection of fault source data by the baseboard management controller after the fault occurs, and realize automatic analysis of fault data and recording of detailed logs by integrating analysis code in the manager, thereby assisting fault diagnosis and fault solving, and avoiding the problem of low efficiency of technical personnel going to the customer site for support after the fault occurs.

[0233] The technical scheme provided by the embodiment of the application can more accurately provide guidance for on-site operation and maintenance by detecting the processor running state through the baseboard management controller, automatically collecting and analyzing processor register information when the system is down, recording detailed fault logs, error content, error reason, maintaining logs and giving diagnosis suggestions.

[0234] Embodiment five

[0235] The application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor can execute the following fault processing method when executing the computer program:

[0236] setting a security network configuration item and an error processing configuration item for the manager port, wherein the security network configuration item is used to prevent changes of a down restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0237] in response to receiving fault alarm information through the manager, obtaining processor register information, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery and fault isolation;

[0238] determining whether to collect fault diagnosis source data according to the current fault diagnosis state;

[0239] in response to collecting fault diagnosis source data, determining whether to perform a soft restart of the system according to the security network configuration item according to the error processing configuration item;

[0240] in response to performing a soft restart of the system according to the security network configuration item, determining a fault type according to the fault diagnosis source data, and automatically analyzing and processing the fault according to the fault type.

[0241] The technical scheme provided by the embodiment of the application has the following beneficial effects:

[0242] The application can realize automatic collection of fault source data by the baseboard management controller after the fault occurs, realize automatic analysis of the fault data and record detailed logs after the automatic analysis of the fault data, assist in fault diagnosis and fault solving, and avoid the problem of low efficiency of technical personnel going to the customer site for support after the fault occurs.

[0243] The technical scheme provided by the embodiment of the application can more accurately provide guidance for on-site operation and maintenance by detecting the processor running state through the baseboard management controller, automatically collecting and analyzing processor register information when the system is down, recording detailed fault logs, error content, error causes, maintaining logs and giving diagnosis suggestions.

[0244] Embodiment six

[0245] The application further provides a computer program product, comprising a computer program which can implement the following fault processing method when executed by a processor:

[0246] setting a security network configuration item and an error processing configuration item for the manager port, wherein the security network configuration item is used to prevent changes of a down restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0247] In response to receiving the fault alarm information through the manager, processor register information is acquired, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery and fault isolation;

[0248] According to the current fault diagnosis state, whether to collect fault diagnosis source data is determined;

[0249] In response to collecting the fault diagnosis source data, whether to perform a soft restart of the system according to the safety network configuration item is determined according to the error processing configuration item;

[0250] In response to performing the soft restart of the system according to the safety network configuration item, a fault type is determined according to the fault diagnosis source data, and the fault is automatically analyzed and processed according to the fault type.

[0251] The technical scheme provided by the embodiment of the application has the following beneficial effects:

[0252] The application can realize automatic collection of fault source data by the baseboard management controller after the fault occurs, realize automatic analysis of the fault data and record detailed logs after the automatic analysis of the fault data, assist in fault diagnosis and fault solving, and avoid the problem of low efficiency of technical personnel going to the customer site for support after the fault occurs.

[0253] The technical scheme provided by the embodiment of the application can more accurately provide guidance for on-site operation and maintenance by detecting the processor running state through the baseboard management controller, automatically collecting and analyzing the processor register information when the system is down, recording detailed fault logs, error content, error reason, maintaining logs and giving diagnosis suggestions.

[0254] Embodiment seven

[0255] The application provides a computer storage medium, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when executing the computer program:

[0256] The safety network configuration item is set for the manager port, and the error processing configuration item is set for the manager port, wherein the safety network configuration item is used to prevent the change of the down restart environment, and the error processing configuration item is used to determine different restart strategies according to error types;

[0257] In response to receiving the fault alarm information through the manager, processor register information is acquired, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery and fault isolation;

[0258] determining whether to collect fault diagnosis source data according to the current fault diagnosis state;

[0259] in response to collecting the fault diagnosis source data, determining whether to perform a soft restart of the system according to the safety network configuration item and the error processing configuration item;

[0260] in response to performing a soft restart of the system according to the safety network configuration item, determining a fault type according to the fault diagnosis source data and performing automatic analysis processing of the fault according to the fault type.

[0261] In one embodiment, before setting the safety network configuration item and the error processing configuration item for the manager port, the method comprises:

[0262] setting a fault diagnosis function in the manager, recording state information when a fault occurs, monitoring the fault alarm information, importing a crash dump error code in the manager, and monitoring the fault diagnosis state;

[0263] starting the safety network configuration item, and when a plurality of flooding attack requests occur in the system, triggering a processor reset and clearing the flooding attack resources;

[0264] starting the error processing configuration item, and when a plurality of fault errors occur in the system, protecting the system by resetting a PCIe controller.

[0265] In one embodiment, the fault diagnosis source data comprises a plurality of fault data blocks, the fault data blocks comprise state data and Internet Protocol identifier data, and the method of determining a fault type according to the fault diagnosis source data and performing automatic analysis processing of the fault according to the fault type comprises:

[0266] determining the validity of the fault data blocks according to the state data;

[0267] in response to the fault data blocks being valid, determining a fault type according to the state data, the Internet Protocol identifier data, and a standard document and performing analysis processing of the fault according to the fault type;

[0268] the analysis processing of the fault according to the fault type comprises:

[0269] analyzing a fault memory location according to the fault type;

[0270] analyzing a fault severity level and a fault code according to the fault type;

[0271] determining a detailed description of the fault according to the fault type;

[0272] According to the fault memory location, the fault severity level, the fault code and the fault detailed description, a fault record fault log is recorded;

[0273] According to the fault log, a fault processing solution is formulated.

[0274] In one of the embodiments, the method comprises:

[0275] In response to no soft restart of the system, a down interface is retained, down processor register information is collected by a collection tool remotely or on site, the down processor register information is parsed, and the fault is processed.

[0276] In one of the embodiments, according to the fault diagnosis source data, the fault type is determined, and according to the fault type, the fault is automatically parsed and processed, which further comprises:

[0277] The fault diagnosis source data is cleaned and formatted to remove data noise, errors and redundant information;

[0278] The fault characteristics of the cleaned fault diagnosis source data are extracted, wherein the fault characteristics include trend analysis, statistical indicators and frequency domain analysis of time series data;

[0279] The fault characteristics are classified by a fault mode training model to determine the fault type;

[0280] According to the fault type, a fault code manual or knowledge base is obtained, and the fault log file is analyzed according to the fault code manual or knowledge base;

[0281] According to the fault log file, the fault is parsed and processed by comparing historical fault cases;

[0282] Whether the fault processing is completed is verified by retesting;

[0283] The verification of whether the fault processing is completed by retesting comprises:

[0284] The conditions when the fault occurs are reproduced to confirm whether the same fault problem occurs;

[0285] The log file of the application program is obtained to confirm whether abnormal records occur;

[0286] The CPU usage, memory usage, disk and network traffic of the system are monitored by a performance monitoring tool to confirm whether the system performance returns to normal;

[0287] The system is stress tested to confirm whether the system can run;

[0288] It is confirmed whether all system configuration files have been modified and saved.

[0289] In one of the embodiments, after determining the fault type according to the fault diagnosis source data, and automatically resolving the fault according to the fault type, the method comprises:

[0290] Recording fault error logs by the manager, and optimizing fault processing according to the fault error logs;

[0291] The optimization of the fault processing according to the fault error logs comprises:

[0292] Obtaining fault error logs;

[0293] Structuring the fault error logs by unifying log formats of the fault error logs;

[0294] Aggregating and statistically analyzing the structured fault error log data to determine fault modes and rules;

[0295] Creating a fault tree according to the fault modes and rules to analyze the fault, and optimizing the fault processing strategy.

[0296] In one of the embodiments, the method comprises:

[0297] When the automatic processing of the fault fails, performing a soft restart of the system, obtaining processor register information, determining a fault error type and a fault component according to the processor register information;

[0298] Creating first fault information according to the fault error type and the fault component information;

[0299] Sending the first fault information to the manager;

[0300] The manager records fault logs according to the first fault information, and resolves the fault logs;

[0301] And replacing the fault component according to the fault component information.

[0302] The application can be compatible with hardware and software, so that after a fault occurs, the manager can automatically collect fault source data to process the fault and improve the correctness of fault processing.

[0303] Figure 6 The exemplary system provided in Embodiment Seven of the application can be used to implement various embodiments described in the application;

[0304] As Figure 6As shown, in some embodiments, the system can be the above-described apparatus for failure handling as any of the described embodiments. In some embodiments, the system can include one or more computer-readable media (e.g., system memory or NVM / storage) having results and one or more processors (e.g., processor(s)) coupled with the one or more computer-readable media and configured to execute the results to implement modules to perform the actions described in this application.

[0305] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, databases, or other media in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0306] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of each technical feature in the above embodiments are not described, however, as long as the combination of these technical features does not exist, it should be considered as the scope of the description.

[0307] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A failure handling method characterized by, The method comprises: setting a security network configuration item and an error handling configuration item for the manager port, wherein the security network configuration item is used to prevent changes in the crash restart environment, and the error handling configuration item is used to determine different restart strategies according to error types; in response to receiving a fault alarm information through the manager, obtaining processor register information, wherein the processor register information comprises a current fault diagnosis state, and the fault diagnosis state comprises at least one of error detection, error reporting, error recovery and fault isolation; determining whether to collect fault diagnosis source data according to the current fault diagnosis state; in response to collecting fault diagnosis source data, determining whether to perform a soft restart of the system according to the security network configuration item according to the error handling configuration item; in response to performing a soft restart of the system according to the security network configuration item, determining a fault type according to the fault diagnosis source data, and automatically analyzing and processing the fault according to the fault type; the fault diagnosis source data comprises a plurality of fault data blocks, the fault data blocks comprise state data and internet protocol identifier data, and the determination of the fault type according to the fault diagnosis source data and the automatic analysis and processing of the fault according to the fault type comprise: judging the validity of the fault data blocks according to the state data; in response to the fault data blocks being valid, determining a fault type according to the state data, the internet protocol identifier data and a standard document, and analyzing and processing the fault according to the fault type; the analysis and processing of the fault according to the fault type comprise: analyzing a fault memory location according to the fault type; analyzing a fault severity level and a fault code according to the fault type; determining a fault detailed description according to the fault type; recording a fault log according to the fault memory location, the fault severity level, the fault code and the fault detailed description; formulating a fault processing solution according to the fault log.

2. The failure handling method according to claim 1, characterized by, Before the setting of the security network configuration item and the error handling configuration item for the manager port, the method comprises: setting a fault diagnosis function in the manager, recording state information when a fault occurs, and monitoring the fault alarm information; importing a crash dump error code in the manager, and monitoring the fault diagnosis state; starting the security network configuration item, and when a plurality of flood attack requests occur in the system, triggering a processor reset and clearing flood attack resources; starting the error handling configuration item, and when a plurality of fault errors occur in the system, protecting the system by resetting a PCIe controller.

3. The failure handling method according to claim 1, characterized by, The method comprises: in response to not performing a soft restart of the system, retaining a crash interface, collecting crash processor register information through a collection tool remotely or on site, analyzing the crash processor register information, and processing the fault.

4. The failure handling method according to claim 1, characterized by, The determination of the fault type according to the fault diagnosis source data and the automatic analysis and processing of the fault according to the fault type further comprise: cleaning and formatting the fault diagnosis source data, and removing data noise, errors and redundant information; The fault feature extraction is performed on the cleaned fault diagnosis source data, wherein the fault feature includes trend analysis, statistical index and frequency domain analysis of time series data; The fault feature is classified by the fault mode training model to determine the fault type; The fault code manual or knowledge base is obtained according to the fault type, and the fault log file is analyzed according to the fault code manual or knowledge base; The fault is analyzed and processed according to the comparison between the fault log file and the historical fault case; The fault processing is verified by retesting; The fault processing is verified by retesting, including: Reappearing the conditions when the fault occurs to confirm whether the same fault problem occurs; Obtaining the log file of the application program to confirm whether abnormal records occur; Monitoring the CPU usage, memory usage, disk and network traffic of the system by the performance monitoring tool to confirm whether the system performance returns to normal; Confirming whether the system can run by performing stress testing on the system; Confirming whether all system configuration files have been modified and saved.

5. The failure handling method according to claim 1, characterized by, After the fault type is determined according to the fault diagnosis source data and the fault is automatically analyzed and processed according to the fault type, including: Recording the fault error log by the manager, and optimizing the fault processing according to the fault error log; The fault processing is optimized according to the fault error log, including: Obtaining the fault error log; Structuring the fault error log by unifying the log format; Aggregating and statistically analyzing the structured fault error log data to determine the fault mode and rule; Creating a fault tree according to the fault mode and rule to analyze the fault and optimize the fault processing strategy.

6. The failure handling method of claim 1, wherein, The method includes: When the fault automatic processing has a problem, the system is soft restarted, the processor register information is obtained, the fault error type and the fault component are determined according to the processor register information; The first fault information is created according to the fault error type and the fault component information; The first fault information is sent to the manager; The manager records the fault log according to the first fault information, and analyzes the fault log; And the fault component is replaced according to the fault component information.

7. A fault handling apparatus for implementing the method of claim 1, wherein, The device includes: A setting module for setting a security network configuration item and an error processing configuration item on the manager port, wherein the security network configuration item is used to prevent the change of the down-restart environment, and the error processing configuration item is used to determine different restart strategies according to the error type; A receiving module for obtaining processor register information in response to receiving fault alarm information through the manager, wherein the processor register information includes a current fault diagnosis state, and the fault diagnosis state includes at least one of error detection, error reporting, error recovery and fault isolation; A judgment module for judging whether to collect fault diagnosis source data according to the current fault diagnosis state; In response to collecting fault diagnosis source data, whether the system is soft restarted according to the security network configuration item is determined according to the error processing configuration item; The processing module is configured to determine a fault type according to the fault diagnosis source data in response to a soft restart of the system according to the security network configuration item, and perform automatic analysis and processing on the fault according to the fault type.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the fault processing method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the fault processing method in any one of claims 1 to 6.

Images