Methods and apparatuses for handling non-recoverable system fault, and electronic device and storage medium

Abstract

The present application relates to the technical field of computers. Provided are methods and apparatuses for handling a non-recoverable system fault, and an electronic device and a non-transitory readable storage medium. A method is applied to a BMC, and comprises: collecting register data of a processor after a fault check signal is detected, wherein the fault check signal is sent to a BMC by an on-chip controller upon detecting that a flag bit of a fault isolation register is set; analyzing the register data to obtain positioning information of a faulty device; and triggering a warm reboot of a system, wherein during the warm reboot of the system, the positioning information is used for instructing a host BIOS to disable the corresponding faulty device on the basis of the positioning information. In the present application, after a non-recoverable system fault is detected, out-of-band collection and analysis are performed on the register data of the processor by means of the BMC, and the processes of data collection and analysis do not occupy processor resources, thereby greatly reducing the workload of the processor, ensuring the processing performance requirements of the processor for a real user service, and improving the stability of a server system.

Description

System unrecoverable fault processing method and device, electronic equipment and storage medium

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 2024108319964, filed on June 26, 2024, and entitled "System unrecoverable fault processing method, device, electronic equipment and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of computer technology, and in particular to a system unrecoverable fault processing method, device, electronic equipment and non-transitory readable storage medium. BACKGROUND

[0004] For a traditional server architecture, for example, an OpenPower architecture server, the implementation of RAS (Reliability, Availability and Serviceability) functions (including fault diagnosis and memory alarm detection and other fault processing processes) is completed by the BIOS (Basic Input / Output System) and the on-chip controller (OCC) of the host end calling the CPU (Central Processing Unit) resources. That is, the entire process of handling the non-recoverable fault (fault collection, analysis and generation of corresponding records) in the traditional OpenPower architecture server is completed by the host end. After the server system occurs a non-recoverable error, the on-chip controller needs to spend a lot of processor resources to collect fault data, analyze fault data and generate result records, which leads to a high processor workload and affects the processing performance of the processor on the real user business. SUMMARY

[0005] The present application provides a system unrecoverable fault processing method, device, electronic equipment and non-transitory readable storage medium, to solve the problem of high processor workload and affecting the processing performance of the processor on the real user business in the prior art system unrecoverable fault processing method.

[0006] The present application provides a system unrecoverable fault processing method applied to a BMC (Baseboard Management Controller), comprising the following steps.

[0007] Collecting register data of the processor after detecting a fault check signal, the fault check signal being sent by the on-chip controller to the BMC after detecting that an identification bit of the fault isolation register is set.

[0008] Analyzing the register data to obtain positioning information of the faulty device.

[0009] Triggering a system hot restart, and during the system hot restart, the positioning information is used to instruct the host BIOS to shield the corresponding faulty device according to the positioning information.

[0010] According to the system non-recoverable fault processing method provided in the application, after detecting a fault check signal, register data of the processor is collected, including: after detecting the fault check signal, register data of registers in a register configuration table is collected according to the register configuration table stored in the BMC, and the register configuration table stores configuration information of registers of the processor.

[0011] According to the system non-recoverable fault processing method provided in the application, register data of registers in a register configuration table is collected according to the register configuration table stored in the BMC, including the following steps.

[0012] Obtaining configuration information in the register configuration table, and the configuration information includes: an offset address of the register, an access mode, a CPU number, a data length, an output position, and a register name.

[0013] Determining a framing format according to the access mode.

[0014] Generating a data frame for accessing the processor according to the framing format, the offset address of the register, the CPU number, and the data length, and sending the data frame as write data of the processor to the processor, and the data frame is used to instruct the processor to read a register value of a register with a length equal to the data length in the register corresponding to the offset address in a case where the CPU number is identical to a number of the processor, and return the register value to the BMC.

[0015] Composing register data in a form of a key-value pair by using the register name and the register value, and generating an intermediate file according to the register data, and writing the intermediate file to an output position designated storage path for storage.

[0016] According to the system non-recoverable fault processing method provided in the application, further including: during the system hot restart, if the fault check signal is detected again, register data of the processor is collected again, an intermediate file is generated again, the intermediate file generated again is merged with the intermediate file generated last time, and for a register for which data has been collected last time, only the register value of the last time is retained.

[0017] According to the system non-recoverable fault processing method provided in the application, after the system hot restart is completed, the method further comprises: deleting the first generated intermediate file or the merged intermediate file.

[0018] According to the system non-recoverable fault processing method provided in the application, the method further comprises: in the case that the number of system hot restarts exceeds the preset restart number and the system startup cannot be completed, prompting that the system cannot be started.

[0019] According to the system non-recoverable fault processing method provided in the application, the method further comprises: the on-chip controller pulls up a checkstop pin connected with the BMC, and the BMC considers that a fault checking signal is detected after detecting that the checkstop pin is pulled up; and the BMC acquires the register data of the processor through an I2C and a replaceable unit interface.

[0020] According to the system non-recoverable fault processing method provided in the application, the register data is analyzed to obtain the positioning information of the fault device, and the method comprises the following steps.

[0021] The first target register value corresponding to the device state is acquired from the register data.

[0022] In the case that the first target register value represents the fault state, the second target register value corresponding to the device address is acquired, the second target register value is the address information of the fault device, and the address information is determined as the positioning information.

[0023] According to the system non-recoverable fault processing method provided in the application, the register data is analyzed to obtain the positioning information of the fault device, and the method comprises the following steps.

[0024] The first target register value corresponding to the device state is acquired from the register data.

[0025] In the case that the first target register value represents the fault state, the second target register value corresponding to the device address is acquired, and the second target register value is the address information of the fault device.

[0026] According to the asset information stored in the BMC, the type of the fault device is determined.

[0027] In the case that the type of the fault device is a PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) device type, the address information of the fault device is converted into the device identifier of the fault device according to the conversion relationship between the address information and the device identifier of the device recorded in the asset information, and the device identifier is determined as the positioning information.

[0028] According to the system non-recoverable fault processing method provided in the application, before triggering the system hot restart, the following step is further included.

[0029] A fault analysis table is generated according to the positioning information of the fault device.

[0030] The fault analysis table is sent to the shared memory for storage, so that the host BIOS obtains the fault analysis table from the shared memory during the system hot restart process.

[0031] According to the system non-recoverable fault processing method provided in the application, a fault analysis table is generated according to the positioning information of the fault device, including the following steps:

[0032] The number of times of failure of the target fault device in the historical fault analysis table is counted, wherein the target fault device is the fault device currently analyzed;

[0033] The number of times is used as the priority of each target fault device;

[0034] The corresponding positioning information is arranged according to the priority of the target fault device, and a fault analysis table is obtained.

[0035] The application further provides a system non-recoverable fault processing method, which is applied to the BIOS of the host side and includes the following steps.

[0036] The state of the identification bit of the fault isolation register is monitored.

[0037] In the case where the identification bit of the fault isolation register is set, the on-chip controller is triggered to send a fault check signal to the BMC, the fault check signal is used to instruct the BMC to collect and analyze the register data of the processor to obtain the positioning information of the fault device, and the system hot restart is triggered.

[0038] During the system hot restart process, the corresponding fault device is shielded according to the positioning information.

[0039] The application further provides a system non-recoverable fault processing method, in the case where the identification bit of the fault isolation register is set, the on-chip controller is triggered to send a fault check signal to the BMC, including the following steps.

[0040] In the case where the identification bit of the fault isolation register is set, an interrupt function corresponding to the system non-recoverable fault is triggered to execute.

[0041] During the execution of the interrupt function, the on-chip controller is triggered to send a fault check signal to the BMC.

[0042] The application further provides a system non-recoverable fault processing method, in the case where the identification bit of the fault isolation register is set, an interrupt function corresponding to the system non-recoverable fault is triggered to execute, including the following steps.

[0043] In a system hot restart process, a fault analysis table is obtained from the shared memory, and the fault analysis table is generated by the BMC according to the positioning information of the faulty device.

[0044] The positioning information is parsed from the fault analysis table, and the corresponding faulty device is shielded according to the positioning information.

[0045] The application also provides a system non-recoverable fault processing method, and the positioning information of each faulty device is identified from the fault analysis table through a special symbol, wherein the positioning information is separated by the special symbol in the fault analysis table.

[0046] The application also provides a system non-recoverable fault processing method, and the application also comprises the following steps.

[0047] The application also provides a system non-recoverable fault processing device, which is applied to the BMC and comprises the following modules.

[0048] The data collection module is configured to collect the register data of the processor after detecting the fault check signal, and the fault check signal is sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set.

[0049] The data analysis module is configured to analyze the register data to obtain the positioning information of the faulty device.

[0050] The restart triggering module is configured to trigger a system hot restart, and in the system hot restart process, the positioning information is used to instruct the host BIOS to shield the corresponding faulty device according to the positioning information.

[0051] The application also provides a system non-recoverable fault processing device, which is applied to the BIOS on the host side and comprises the following modules.

[0052] The fault detection module is configured to monitor the state of the identification bit of the fault isolation register.

[0053] The fault check signal sending module is configured to trigger the on-chip controller to send the fault check signal to the BMC in the case that the identification bit of the fault isolation register is set, and the fault check signal is used to instruct the BMC to collect and analyze the register data of the processor to obtain the positioning information of the faulty device, and trigger a system hot restart.

[0054] The fault device shielding module is configured to shield the corresponding fault device according to the positioning information during a system hot restart.

[0055] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the system unrecoverable fault processing method applied to the BMC or the system unrecoverable fault processing method applied to the BIOS of the host side when executing the program.

[0056] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the system unrecoverable fault processing method applied to the BMC or the system unrecoverable fault processing method applied to the BIOS of the host side.

[0057] The system unrecoverable fault processing method, device, electronic device, and non-transitory computer readable storage medium provided by the application, by collecting register data of the processor by the BMC after detecting a fault checking signal, the fault checking signal is sent to the BMC by the on-chip controller after detecting that the identification bit of the fault isolation register is set; and analyzing the register data by the BMC to obtain positioning information of the fault device; triggering a system hot restart, and the positioning information is used to instruct the host BIOS to shield the corresponding fault device according to the positioning information during the system hot restart. In the application, after detecting the system unrecoverable fault, the register data of the processor is collected and analyzed by the BMC out of band, and the data collection and analysis process does not occupy the processor resources, greatly reducing the work load of the processor, ensuring the processing performance requirement of the processor to the real user service, and improving the stability of the server system. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Fig. 1 is a schematic diagram of a system hardware structure used in the system unrecoverable fault processing method provided by the application.

[0060] Fig. 2 is a flowchart of the system unrecoverable fault processing method provided by the application.

[0061] Fig. 3 is a flowchart of the system unrecoverable fault processing method provided by the application.

[0062] Figure 4 is a third flow diagram of a method for handling a system unrecoverable failure according to the present application.

[0063] Figure 5 is a fourth flow diagram of a method for handling a system unrecoverable failure according to the present application.

[0064] Figure 6 is a fifth flow diagram of a method for handling a system unrecoverable failure according to the present application.

[0065] Figure 7 is an interaction diagram of a processor, a BIOS and a BMC in a method for handling a system unrecoverable failure according to the present application.

[0066] Figure 8 is a first structural diagram of a system for handling a system unrecoverable failure according to the present application.

[0067] Figure 9 is a second structural diagram of a system for handling a system unrecoverable failure according to the present application.

[0068] Figure 10 is a structural diagram of an electronic device according to the present application. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0070] As shown in FIG. 1, it is a schematic diagram of openpower server hardware structure. For the traditional RAS detection function, when the system hardware error checker detects the non-recoverable error of the system, the identification bit of the fault isolation register (FIR) is set. After the BIOS detects that the identification bit of the fault isolation register is set, the on-chip controller (OCC) is triggered to collect fault data and save the fault data into the shared memory (flash) of the system, and sends a fault check signal (i.e. checkstop signal) to the baseboard management controller (BMC). When the BMC detects the checkstop signal, the system is triggered to hot restart. After the system hot restarts, the BIOS reads and analyzes the fault data, generates a RAS record, and isolates the abnormal component according to the RAS record, and then triggers the system to hot restart. Since the abnormal component has been isolated, the abnormal component will not be started again after the system hot restarts, and the system can run normally. Both the fault data collection of the on-chip controller and the fault data analysis of the BIOS need to occupy the processor (CPU) resources, especially in the fault data collection stage, which will increase the workload of the CPU and affect the processing performance of the normal real user business.

[0071] Based on the above technical problem, the system non-recoverable fault processing method of the embodiment of the present application, as shown in FIGS. 1 and 2, includes the following steps S210 to S230.

[0072] Step S210: collecting the register data of the processor after detecting the fault check signal. The fault check signal is sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set. Optionally, during the running of the server system, when the system hardware error checker detects the non-recoverable error of the system, the identification bit of the fault isolation register is set. After the BIOS detects that the identification bit of the fault isolation register is set, the on-chip controller (OCC) sends the fault check signal to the BMC, as shown in FIG. 1, i.e. the on-chip controller (OCC) pulls up the checkstop pin connected with the BMC. After the BMC detects that the checkstop pin is pulled up, it is considered that the fault check signal is detected, and the register data of the processor is collected. Since the state and address information of each device in the system is recorded in the register of the processor, the state of each device can be obtained by collecting the register data of the processor, so as to analyze whether it is a fault device.

[0073] Step S220: analyzing the register data to obtain the positioning information of the fault device. Optionally, the BMC can analyze whether the device is faulty according to the state of each device recorded in the register of the processor, and directly take the address information as the positioning information, so as to obtain the positioning information of the fault device.

[0074] Step S230: triggering a system hot restart, and in the system hot restart process, the positioning information is used to instruct the host BIOS to shield the corresponding faulty device according to the positioning information. The BMC triggers the system hot restart after collecting and analyzing the register data of the processor, so that in the system hot restart process, the BIOS shields the corresponding faulty device according to the positioning information, so as to ensure that the server system can be started normally.

[0075] In the system unrecoverable fault processing method of the embodiment, the register data of the processor is collected by the BMC, and the register data is analyzed by the BMC to obtain the positioning information of the faulty device; the system hot restart is triggered, so that in the system hot restart process, the BIOS on the host side shields the corresponding faulty device according to the positioning information. That is, after detecting the system unrecoverable fault, the register data of the processor is collected and analyzed by the BMC out of band, and the data collection and analysis process of the RAS detection function does not occupy the processor resources, greatly reducing the working load of the processor, ensuring the processing performance requirements of the processor to the real user service, and improving the stability of the server system. In addition, the register data is collected and analyzed by the BMC out of band, which avoids the problem of failure of the faulty component isolation caused by the fact that the fault data cannot be completely collected successfully due to the on-chip controller OCC itself exception in the traditional scheme.

[0076] It should be noted that the system unrecoverable fault processing method of the embodiment is not limited to be applied to the openpower server system, and can also be used in other server systems.

[0077] In some embodiments, step S210 includes: after detecting the fault check signal, collecting the register data of the registers in the register configuration table according to the register configuration table stored in the BMC, and the register configuration table is configured with the configuration information of the registers of the processor. As shown in FIG. 1, since the BMC is an out-of-band device, it needs to interact with the processor through I2C (Inter-Integrated Circuit, Inter-Integrated Circuit) and Field Replaceable Unit Service Interface (FSI). In this embodiment, the register data of the processor can be obtained through I2C and the Field Replaceable Unit Service Interface. Moreover, the register configuration table stored in the BMC in advance is configured with the configuration information of the registers of the processor, so that the BMC collects the data of the registers recording the states and addresses of the devices in the system purposefully, instead of collecting all the register data of the processor, thereby improving the data collection efficiency.

[0078] In some embodiments, as shown in FIG. 3, collecting register data of registers in the register configuration table according to the register configuration table stored in the BMC includes the following steps S310-S340.

[0079] Step S310: Obtain configuration information in the register configuration table, the configuration information including: offset address of the register, access mode, CPU number, data length, output location and register name. Wherein, the offset address can uniquely locate a register, the access mode is the access mode of the register, different registers have different access modes, the CPU number indicates which CPU the register belongs to, the data length is the length of the register value, and the output location is the saving location directory of the obtained register value. For example, the configuration information in the register configuration table is as follows.

[0080] "OutputPath":"Collection_data / cpu%d_bankinfo".

[0081] "rdmcinfo":"CpuBankInfo".

[0082] {"Params":["Target",1,"0x123",8],"Output":{"Size":8,"Name":["mc1_status","MC1"]}}.

[0083] {"Params":["Target",1,"0x124",8],"Output":{"Size":8,"Name":["mc1_addr","MC1"]}}.

[0084] {"Params":["Target",1,"0x125",8],"Output":{"Size":8,"Name":["mc1_misc","MC1"]}}.

[0085] {"Params":["Target",2,"0x123",8],"Output":{"Size":8,"Name":["mc2_status","MC2"]}}.

[0086] {"Params":["Target",2,"0x124",8],"Output":{"Size":8,"Name":["mc2_addr","MC2"]}}.

[0087] {"Params":["Target",2,"0x125",8],"Output":{"Size":8,"Name":["mc2_misc","MC2"]}}.

[0088] Wherein, the OutputPath represents the output position, the rdmcinfo represents the access mode, the 1 represents the CPU number, the 0x123 represents the offset address, the 8 represents the data length, the mc1_status represents the register name, the MC1 represents the register value variable, and the CpuBankInfo represents the bank information of the CPU.

[0089] Step S320: Determine the framing format according to the access mode, different registers have different access modes, for example: the register accessing the CPU basic information adopts the ping access mode, the register accessing the bank information of the CPU adopts the rdmcinfo access mode, and different access modes have different framing formats.

[0090] Step S330: According to the offset address, the CPU number and the data length of the register, generate a data frame for accessing the processor according to the framing format, and send the data frame to the processor as the write data of the processor, the data frame is used to instruct the processor to read the register value of the register corresponding to the offset address in the case of confirming that the CPU number is the same as the number of itself, and return the register value to the BMC. Optionally, taking the register accessing the bank information of the CPU as an example, the data frame composed according to the framing format corresponding to the rdmcinfo access mode is: i2ctest w1@0x30 0x123 r8, which means that the CPU with the CPU number of 1 and the address of 0x30 is written(w) by I2C to read the register with the address offset of 0x123, and the register value with the data length of 8 is read(r), the register value corresponds to the register name of mc1_status, and the mc1_status represents the device status. Similarly, the register value of the register with the register name of mc1_addr can also be obtained, and the mc1_addr represents the device address information.

[0091] Step S340: The register name and the register value are composed into the register data in the form of key-value pair, and the intermediate file is generated according to the register data, and the intermediate file is written into the output position designated storage path for saving. For example, the mc1_status and the corresponding register value, and the mc1_addr and the corresponding register value are composed into the register data.

[0092] In this embodiment, on the basis of the register configuration table, the BMC can efficiently collect the data collection process of the RAS detection function through the above four steps.

[0093] In some embodiments, the system unrecoverable failure processing method further comprises: during the system hot restart, if the fault check signal is detected again, the register data of the processor is collected again, the intermediate file is generated again, the intermediate file generated again is merged with the intermediate file generated last time, and for the registers whose data has been collected last time, only the register value of last time is kept. If the fault check signal is detected again during the system hot restart, it means that the shielding of the BIOS to the faulty device is unsuccessful, and it is also possible that some register data is not collected due to I2C transmission during the first data collection. Therefore, if the fault check signal is detected again during the system hot restart, the register data of the processor is collected again, and the intermediate file is generated again. For the registers whose data has been collected last time, since the register data collected last time is closer to the time when the first unrecoverable failure occurs, the collected register data can better reflect the actual situation of the unrecoverable failure, therefore, by comparing the intermediate files corresponding to the two data collections, for the registers whose data has been collected last time, only the register value of last time is kept, that is, the register value collected again is discarded.

[0094] In some embodiments, the system unrecoverable failure processing method further comprises: after the system hot restart is completed, the intermediate file generated first time or the merged intermediate file is deleted. In this embodiment, the completion of the system hot restart means that the devices causing the system unrecoverable failure have all been successfully shielded, and the intermediate file generated first time or the merged intermediate file is deleted to avoid the conflict between the register data collected during the system normal operation and the intermediate file collected during the first unrecoverable failure, and to save the flash memory space of the system.

[0095] In some embodiments, the system unrecoverable failure processing method further comprises: in the case that the number of system hot restarts exceeds the preset number of restarts and the system startup cannot be completed, prompting that the system cannot be started, that is, prompting the maintenance personnel that the faulty device cannot be shielded and manual troubleshooting is required.

[0096] In some embodiments, the process of step S220 is shown in FIG. 4, which includes the following steps S410 to S420.

[0097] Step S410: obtaining a first target register value in the register data corresponding to the device state, the first target register value representing the device state, which can be used to identify whether the device is faulty. For RAS failure, a set of RAS diagnosis rules is sorted out by BIOS engineers according to the characteristics of RAS failure, for example: the CPU bank state information in RAS is used to infer whether the CPU error or the memory error.

[0098] Step S420: In the case that the first target register value represents a fault state, the second target register value corresponding to the register name of the device address is obtained, the second target register value is address information of the fault device, and the address information is determined as the positioning information. For example, the register value corresponding to the register mc1_status represents the device state, and the register value corresponding to the register mc1_addr corresponds to the address information of the device, so that the address information of the fault device can be obtained.

[0099] In this embodiment, the BMC can quickly obtain the address information of the fault device by analyzing the register values representing the device state and the device address in the register data, and directly uses the address information as the positioning information, so that the BIOS can shield the corresponding fault device according to the address information.

[0100] In some embodiments, the step S220 includes the following steps S510-S540, as shown in FIG. 5.

[0101] Step S510: Obtain the first target register value corresponding to the register name of the device state in the register data.

[0102] Step S520: In the case that the first target register value represents a fault state, the second target register value corresponding to the register name of the device address is obtained, the second target register value is address information of the fault device, and the address information is determined as the positioning information. For example, the register value corresponding to the register mc1_status represents the device state, and the register value corresponding to the register mc1_addr corresponds to the address information of the device, so that the address information of the fault device can be obtained.

[0103] Step S530: Determine the type of the fault device according to the asset information stored in the BMC. Optionally, the asset information includes the address, type and identifier of all devices in the system, and the system automatically loads the asset information into the BMC at the start of the boot.

[0104] Step S540: In the case that the type of the fault device is a PCIe device type, the address information of the fault device is converted into the device identifier of the fault device according to the conversion relationship between the address information and the device identifier of the device recorded in the asset information, and the device identifier is determined as the positioning information. For a PCIe device, the device identifier is BDF (Bus, Device, Function), and the system usually locates and accesses the PCIe device through the BDF. Therefore, if the fault device is a PCIe device, the BMC converts the address information of the fault device into the device identifier of the fault device according to the conversion relationship between the address information and the device identifier of the device recorded in the asset information, so that the BIOS can directly shield the corresponding PCIe device through the device identifier, avoiding the conversion of the address information and the device identifier in the BIOS and reducing the processor resource occupation.

[0105] In some embodiments, before triggering the system hot restart, the method further comprises: generating a fault analysis table according to the location information of the faulty device; and sending the fault analysis table to the shared memory for storage, so that during the system hot restart, the host BIOS obtains the fault analysis table from the shared memory. The shared memory is a flash memory, and data can be saved during system restart, so that the BIOS can also obtain the fault analysis table after restart. Optionally, the fault analysis table can be generated according to the location information of the faulty device in a predetermined format, for example, the location information of each faulty device is separated by a special symbol, and the BIOS parses the fault analysis table according to the predetermined format to obtain the location information of each faulty device.

[0106] Optionally, the step of generating the fault analysis table according to the location information of the faulty device is as follows: taking each analyzed faulty device as a target faulty device, for each target faulty device, counting the number of times of failure of the target faulty device in the historical fault analysis table, taking the number of times as the priority of each target faulty device, and arranging the corresponding location information of the target faulty device in the currently generated fault analysis table according to the priority of the target faulty device, i.e., arranging the location information of the target faulty device with high priority in front and arranging the location information of the target faulty device with low priority at the back. When the BIOS reads the fault analysis table, the target faulty device corresponding to the location information read first has a higher priority, which is shielded first to avoid system downtime or other device failure caused by late shielding.

[0107] As shown in FIG. 6, the system unrecoverable fault processing method of the embodiment of the application is applied to the BIOS of the host side, and includes the following steps S610 to S630.

[0108] Step S610: monitoring the state of the identification bit of the fault isolation register. Optionally, during the running of the server system, when the system hardware error checker detects an unrecoverable error of the system, the identification bit of the fault isolation register is set, and the BIOS monitors the state of the identification bit of the fault isolation register.

[0109] Step S620: in the case where the identification bit of the fault isolation register is set, triggering the on-chip controller to send a fault checking signal to the BMC, the fault checking signal being used to instruct the BMC to collect and analyze the register data of the processor to obtain the location information of the faulty device, and triggering the system hot restart.

[0110] Step S630: during the system hot restart, shielding the corresponding faulty device according to the location information.

[0111] In the system unrecoverable failure processing method of the embodiment, after the BIOS on the host side detects the system unrecoverable failure, the BMC is triggered to collect and analyze the register data of the processor out of band, and the data collection and analysis process does not occupy the processor resources, greatly reduces the working load of the processor, guarantees the processing performance requirement of the processor on the real user service, and improves the stability of the server system.

[0112] In some embodiments, step S620 includes: in the case that the identification bit of the failure isolation register is set, triggering an interrupt function corresponding to the system unrecoverable failure to be executed; and triggering the on-chip controller to send a failure check signal to the BMC when the interrupt function is executed. Optionally, the failure isolation register is configured as an interrupt source in the BIOS startup process, and the interrupt processing function is triggered after the state of the failure isolation register is abnormal, and the function of triggering the on-chip controller to send the failure check signal to the BMC is implemented in the interrupt function, so that the BMC is informed of the failure check signal in real time when each unrecoverable failure is monitored, and the BMC executes the corresponding data collection and analysis process in time.

[0113] In some embodiments, step S630 includes: in the system hot restart process, the BIOS acquires the failure analysis table from the shared memory, and the failure analysis table is generated by the BMC according to the positioning information of the failed device; the BIOS parses the positioning information from the failure analysis table, and shields the corresponding failed device according to the positioning information. Optionally, the failure analysis table can be parsed according to the corresponding format of generating the failure analysis table, for example: the positioning information of each failed device is separated by a special symbol, and the BIOS parses the positioning information of each failed device by recognizing the special symbol when parsing the failure analysis table.

[0114] In an optional embodiment, as shown in FIG. 7, during the normal operation of the server system, the BIOS detects the failure, that is, detects that the identification bit of the failure isolation register is set, and then triggers the on-chip controller OCC to send a failure check signal to the BMC. After receiving the failure check signal, the BMC performs a data collection operation to collect the register data of the CPU. After the data collection is completed, the BMC performs data analysis on the register data to obtain the positioning information of the failed device, generates a failure analysis table according to the positioning information of the failed device, stores the failure analysis table in the shared memory of the system, so as to be acquired by the BIOS, and then triggers the system to be hot restarted. During the system hot restart process, the BIOS locates the failed device according to the positioning information in the failure analysis table, and shields the failed device.

[0115] As can be seen from the above process, the collection and analysis of the register data of the CPU are performed by the BMC out of band, the BIOS only calls the CPU resource in the stage of detecting the occurrence of the fault and finally shielding the faulty device, and the shielding of the faulty device is in the process of hot restart, so that the non-recoverable fault handling method of the system of the embodiment greatly reduces the work load of the processor, guarantees the processing performance requirement of the processor on the real user service, and improves the stability of the server system.

[0116] The non-recoverable fault handling device provided by the application is described below, and the non-recoverable fault handling device described below can be correspondingly referred to the non-recoverable fault handling method described above.

[0117] The embodiment of the application further provides a non-recoverable fault handling device of a system, as shown in FIG. 8, applied to a BMC, and the device comprises the following modules.

[0118] The data collection module 810 is configured to collect the register data of the processor after detecting the fault checking signal, the fault checking signal being sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set.

[0119] The data analysis module 820 is configured to analyze the register data to obtain the positioning information of the faulty device.

[0120] The restart triggering module 830 is configured to trigger the hot restart of the system, and in the process of the hot restart of the system, the positioning information is used to instruct the host BIOS to shield the corresponding faulty device according to the positioning information.

[0121] In the non-recoverable fault handling device of the system of the embodiment of the application, the register data of the processor is collected and analyzed out of band by the BMC after detecting the non-recoverable fault of the system, the data collection and analysis process does not occupy the processor resource, greatly reduces the work load of the processor, guarantees the processing performance requirement of the processor on the real user service, and improves the stability of the server system.

[0122] In some embodiments, the data collection module 810 is configured to collect the register data of the registers in the register configuration table according to the register configuration table stored in the BMC after detecting the fault checking signal, and the register configuration table is configured with the configuration information of the registers of the processor.

[0123] In some embodiments, the data collection module 810 comprises the following modules.

[0124] The configuration information acquisition module is configured to acquire the configuration information in the register configuration table, and the configuration information comprises the offset address, the access mode, the CPU number, the data length, the output position and the register name of the register.

[0125] The framing format determination module is configured to determine the framing format according to the access mode.

[0126] The data frame generation module is configured to generate a data frame for accessing the processor according to the offset address of the register, the CPU number and the data length in the framing format, and send the data frame as write data of the processor to the processor, the data frame being used to instruct the processor to read a register value of a register with a length equal to the data length in the register corresponding to the offset address and return the register value to the BMC, in a case where the CPU number is identical to the number of the processor.

[0127] The intermediate file generation module is configured to compose register data in the form of key-value pairs by using the register name and the register value, and generate an intermediate file according to the register data, and write the intermediate file into the output location designated storage path for storage.

[0128] In some embodiments, the system unrecoverable fault processing apparatus further comprises an intermediate file merging module configured to, in the process of the system hot restart, if the fault checking signal is detected again, collect register data of the processor again, generate an intermediate file again, merge the intermediate file generated again with the intermediate file generated last time, and for a register for which data has been collected last time, only retain the register value of last time as the corresponding register value.

[0129] In some embodiments, the system unrecoverable fault processing apparatus further comprises an intermediate file deletion module configured to, after the system hot restart is completed, delete the intermediate file generated first time or the intermediate file after merging.

[0130] In some embodiments, the system unrecoverable fault processing apparatus further comprises a restart number limitation module configured to, in a case where the number of system hot restarts exceeds a preset restart number and the system startup cannot be completed, prompt that the system cannot be started.

[0131] In some embodiments, the data analysis module 820 comprises the following modules.

[0132] The first target register value acquisition module is configured to acquire a first target register value corresponding to the device state in the register data.

[0133] The second target register value acquisition module is configured to, in a case where the first target register value represents a fault state, acquire a second target register value corresponding to the device address in the register data, the second target register value being address information of the faulty device, and determine the address information as the positioning information.

[0134] In some embodiments, the data analysis module 820 comprises the following modules.

[0135] The first target register value obtaining module is configured to obtain a first target register value corresponding to the device state in the register data.

[0136] The second target register value obtaining module is configured to obtain a second target register value corresponding to the device address in the register data in a case where the first target register value represents a fault state, the second target register value being address information of the fault device.

[0137] The device type determining module is configured to determine the type of the fault device according to asset information stored in the BMC.

[0138] The device identifier converting module is configured to convert the address information of the fault device into a device identifier of the fault device according to a conversion relationship between the address information of the device and the device identifier recorded in the asset information in a case where the type of the fault device is a PCIe device type, the device identifier being the positioning information.

[0139] In some embodiments, the system unrecoverable fault processing apparatus further comprises the following modules.

[0140] The fault analysis table generating module is configured to generate a fault analysis table according to the positioning information of the fault device before triggering the system hot restart.

[0141] The fault analysis table storing module is configured to store the fault analysis table in the shared memory, so that the host BIOS obtains the fault analysis table from the shared memory during the system hot restart.

[0142] Embodiments of the present application also provide a system unrecoverable fault processing apparatus, as shown in FIG. 9, applied to a BIOS on a host side, the apparatus comprising the following modules.

[0143] The fault detection module 910 is configured to monitor a state of an identification bit of a fault isolation register.

[0144] The fault check signal sending module 920 is configured to trigger the on-chip controller to send a fault check signal to the BMC in a case where the identification bit of the fault isolation register is set, the fault check signal being used to instruct the BMC to collect and analyze register data of the processor to obtain positioning information of the fault device, and trigger a system hot restart.

[0145] The fault device shielding module 930 is configured to shield the corresponding fault device according to the positioning information during the system hot restart.

[0146] In the system unrecoverable failure processing apparatus of the embodiment of the present application, after the BIOS on the host side detects a system unrecoverable failure, the BMC is triggered to collect and analyze the register data of the processor out of band, and the data collection and analysis process does not occupy the processor resources, greatly reducing the working load of the processor, guaranteeing the processing performance requirement of the processor on the real user service, and improving the stability of the server system.

[0147] In some embodiments, the failure checking signal sending module 920 is configured to trigger the execution of an interrupt function corresponding to the system unrecoverable failure in the case where the identification bit of the failure isolation register is set; and trigger the on-chip controller to send a failure checking signal to the BMC when the interrupt function is executed.

[0148] In some embodiments, the failure device shielding module 930 is configured to acquire a failure analysis table from the shared memory in the system hot restart process, the failure analysis table being generated by the BMC according to the positioning information of the failure device; and parse the positioning information from the failure analysis table and shield the corresponding failure device according to the positioning information.

[0149] FIG. 10 illustrates an entity structure diagram of an electronic device, as shown in FIG. 10, the electronic device can include a processor 1010, a communications interface 1020, a memory 1030 and a communications bus 1040, wherein the processor 1010, the communications interface 1020 and the memory 1030 complete mutual communication through the communications bus 1040. The processor 1010 can invoke the logical instructions in the memory 1030 to execute the system unrecoverable failure processing method for the BMC, the method comprising: collecting register data of the processor after detecting a failure checking signal, the failure checking signal being sent by the on-chip controller to the BMC after detecting that the identification bit of the failure isolation register is set; analyzing the register data to obtain positioning information of a failure device; and triggering a system hot restart, wherein the positioning information is used to instruct the host BIOS to shield the corresponding failure device according to the positioning information in the system hot restart process.

[0150] Alternatively, a system unrecoverable failure processing method for the BIOS on the host side is executed, the method comprising: monitoring the state of the identification bit of the failure isolation register; triggering the on-chip controller to send a failure checking signal to the BMC in the case where the identification bit of the failure isolation register is set, the failure checking signal being used to instruct the BMC to collect and analyze the register data of the processor to obtain positioning information of a failure device, and trigger a system hot restart; and shielding the corresponding failure device according to the positioning information in the system hot restart process.

[0151] Further, the logic instructions in the memory 1030 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a non-transitory computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a non-transitory readable storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned non-transitory readable storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] In another aspect, the present application also provides a computer program product, the computer program product comprising a computer program, the computer program being stored in a non-transitory computer readable storage medium, and the computer program being executed by a processor, so that the computer can execute the system unrecoverable fault handling method for the BMC provided by the above-mentioned method, the method comprising: collecting register data of the processor after detecting a fault check signal, the fault check signal being sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set; analyzing the register data to obtain positioning information of the fault device; triggering a system hot restart, and during the system hot restart, the positioning information is used to instruct the host BIOS to shield the corresponding fault device according to the positioning information.

[0153] Alternatively, a system unrecoverable fault handling method for the host side BIOS is executed, the method comprising: monitoring the state of the identification bit of the fault isolation register; in the case that the identification bit of the fault isolation register is set, triggering the on-chip controller to send a fault check signal to the BMC, the fault check signal being used to instruct the BMC to collect and analyze register data of the processor to obtain positioning information of the fault device, and trigger a system hot restart; and during the system hot restart, shielding the corresponding fault device according to the positioning information.

[0154] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the system non-recoverable fault handling method for BMC provided by the above method, the method comprising: collecting register data of the processor after detecting a fault check signal, the fault check signal being sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set; analyzing the register data to obtain positioning information of the faulty device; triggering a system hot restart, and during the system hot restart, the positioning information is used to instruct the host BIOS to shield the corresponding faulty device according to the positioning information.

[0155] Alternatively, a system non-recoverable fault handling method for the host side BIOS is executed, the method comprising: monitoring the state of the identification bit of the fault isolation register; in the case where the identification bit of the fault isolation register is set, triggering the on-chip controller to send a fault check signal to the BMC, the fault check signal being used to instruct the BMC to collect and analyze register data of the processor to obtain positioning information of the faulty device, and trigger a system hot restart; and during the system hot restart, shielding the corresponding faulty device according to the positioning information.

[0156] The device embodiments described above are merely illustrative, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0157] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a non-transitory computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the various embodiments or some part of the embodiments.

[0158] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of handling non-recoverable system failure, characterized by, The application is applied to a baseboard management controller (BMC) and comprises the following steps: Collecting register data of a processor after detecting a fault check signal, which is sent by an on-chip controller to a BMC after detecting that an identification bit of a fault isolation register is set; Analyzing the register data to obtain positioning information of a faulty device; Triggering a system hot restart, and using the positioning information to instruct a host basic input / output system (BIOS) to shield a corresponding faulty device during the system hot restart.

2. The system non-recoverable failure handling method of claim 1, wherein, The step of collecting the register data of the processor after detecting the fault check signal comprises the following steps: After detecting the fault check signal, collecting register data of registers in a register configuration table according to the register configuration table stored in the BMC, wherein configuration information of registers of the processor is configured in the register configuration table.

3. The system non-recoverable failure handling method of claim 2, wherein, The step of collecting the register data of the registers in the register configuration table according to the register configuration table stored in the BMC comprises the following steps: Obtaining the configuration information in the register configuration table, wherein the configuration information comprises an offset address of a register, an access mode, a central processing unit (CPU) number, a data length, an output position and a register name; Determining a framing format according to the access mode; Generating a data frame for accessing the processor according to the offset address of the register, the CPU number and the data length in the framing format, and sending the data frame as write data of the processor to the processor, wherein the data frame is used to instruct the processor to read a register value of a register with a length equal to the data length in a register corresponding to the offset address and return the register value to the BMC, in a case where the CPU number is identical to a number of the processor; Assembling the register data in the form of a key-value pair by using the register name and the register value, and generating an intermediate file according to the register data, and saving the intermediate file in a storage path specified by the output position.

4. The system non-recoverable failure handling method of claim 3, wherein, The application further comprises the following steps: During the system hot restart, if the fault check signal is detected again, the register data of the processor is collected again, an intermediate file is generated again, the intermediate file generated again is combined with the intermediate file generated last time, and for a register whose data has been collected last time, only the register value of the last time is retained.

5. The system non-recoverable failure handling method of claim 4, wherein, After the system hot restart is completed, the first generated intermediate file or the combined intermediate file is deleted.

6. The system non-recoverable failure handling method of claim 4, wherein, The application further comprises the following steps: In a case where the number of system hot restarts exceeds a preset number of restarts and the system cannot be started, it is prompted that the system cannot be started.

7. The system non-recoverable failure handling method of claim 1, wherein, The on-chip controller pulls up a checkstop pin connected to the BMC, the BMC considers that the fault check signal is detected after detecting that the checkstop pin is pulled up, and the BMC obtains the register data of the processor through an I2C and a replaceable unit interface.

8. The system non-recoverable failure handling method of claim 1, wherein, The step of analyzing the register data to obtain positioning information of a faulty device comprises the following steps: Obtaining a first target register value corresponding to a device state in the register name in the register data; In a case where the first target register value represents a fault state, a second target register value corresponding to a device address is obtained, and the second target register value is address information of the fault device.

9. The system non-recoverable failure handling method of claim 1, wherein, The register data is analyzed to obtain positioning information of the fault device, including: a first target register value corresponding to a device state in the register data is obtained; in a case where the first target register value represents a fault state, a second target register value corresponding to a device address is obtained, and the second target register value is address information of the fault device; According to the asset information stored in the BMC, the type of the fault device is determined; in a case where the type of the fault device is a PCIe device type, the address information of the fault device is converted into a device identifier of the fault device according to the conversion relationship between the address information and the device identifier recorded in the asset information, and the device identifier is determined as the positioning information.

10. The system non-recoverable failure handling method according to any one of claims 1 to 9, characterized in that, Before triggering a system hot restart, further comprising: generating a fault analysis table according to the positioning information of the fault device; sending the fault analysis table to a shared memory for storage, so that the host BIOS obtains the fault analysis table from the shared memory during the system hot restart process.

11. The system non-recoverable failure handling method of claim 10, wherein, The fault analysis table is generated according to the positioning information of the fault device, including: counting the number of times of failure of the target fault device in the historical fault analysis table, wherein the target fault device is the fault device currently analyzed; the priority of each target fault device is determined according to the number of times; the corresponding positioning information is arranged according to the priority of the target fault device to obtain the fault analysis table.

12. A system non-recoverable failure handling method, characterized by, The BIOS applied to the host side, including: monitoring the state of the identification bit of the fault isolation register; in a case where the identification bit of the fault isolation register is set, triggering the on-chip controller to send a fault check signal to the BMC, and the fault check signal is used to instruct the BMC to collect and analyze the register data of the processor to obtain the positioning information of the fault device, and trigger a system hot restart; during the system hot restart process, the corresponding fault device is shielded according to the positioning information.

13. The system non-recoverable failure handling method of claim 12, wherein, in a case where the identification bit of the fault isolation register is set, triggering the on-chip controller to send a fault check signal to the BMC, including: in a case where the identification bit of the fault isolation register is set, triggering the execution of the interrupt function corresponding to the system non-recoverable fault; when the interrupt function is executed, the on-chip controller sends the fault check signal to the BMC.

14. The system non-recoverable failure handling method according to claim 12 or 13, characterized by, During the system hot restart process, the corresponding fault device is shielded according to the positioning information, including: during the system hot restart process, the fault analysis table is obtained from the shared memory, and the fault analysis table is generated by the BMC according to the positioning information of the fault device; the positioning information is parsed from the fault analysis table, and the corresponding fault device is shielded according to the positioning information.

15. The system non-recoverable failure handling method of claim 14, wherein, The resolving of the positioning information from the fault analysis table comprises: identifying the positioning information of each fault device from the fault analysis table through special symbols, wherein the positioning information is separated by the special symbols in the fault analysis table.

16. The system non-recoverable failure handling method of claim 12, wherein, The on-chip controller pulls up a checkstop pin connected to the BMC, and the BMC considers that the fault check signal is detected when the checkstop pin is pulled up.

17. A system non-recoverable failure handling apparatus characterized by comprising: The application is applied to the BMC, and comprises: The data collection module is configured to collect register data of the processor after detecting the fault check signal, wherein the fault check signal is sent by the on-chip controller to the BMC after detecting that the identification bit of the fault isolation register is set. The data analysis module is configured to analyze the register data to obtain positioning information of a fault device. The restart triggering module is configured to trigger a system hot restart, and the positioning information is used to instruct the host BIOS to shield the corresponding fault device according to the positioning information during the system hot restart.

18. A system non-recoverable failure handling apparatus characterized by comprising: The application is applied to the BIOS on the host side, and comprises: The fault detection module is configured to monitor the state of the identification bit of the fault isolation register. The fault check signal sending module is configured to trigger the on-chip controller to send the fault check signal to the BMC when the identification bit of the fault isolation register is set, wherein the fault check signal is used to instruct the BMC to collect and analyze the register data of the processor to obtain the positioning information of the fault device, and trigger the system hot restart. The fault device shielding module is configured to shield the corresponding fault device according to the positioning information during the system hot restart.

19. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the program to implement the system unrecoverable fault processing method according to any one of claims 1 to 10, or implement the system unrecoverable fault processing method according to any one of claims 12 to 14. 20.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by the processor to implement the system unrecoverable fault processing method according to any one of claims 1 to 10, or implement the system unrecoverable fault processing method according to any one of claims 12 to 14.

Images