Method for testing server system through automated script and related product

Abstract

The invention discloses a method for testing a server system through an automatic script and a related product. The method provided by the invention comprises the following steps of: injecting simulation abnormity into a server system in a power-on self-test stage so as to trigger the server system to be down; and judging whether a secondary fault recovery starting mechanism of the server system is normal or not according to the condition that the server system executes the recovery operation. According to the method provided by the invention, a test code is written in an automatic script, simulation abnormity is injected in a POST stage after the script runs so as to trigger downtime, and whether the function of an FRB-2 mechanism is normal or not is judged by detecting the condition that a server system executes recovery operation after downtime. Therefore, the server system can automatically execute the FRB-2 mechanism test item, so that the dependence on manpower is reduced, and the efficiency and accuracy of the test work can be improved.

Description

Technical Field

[0001] This application generally relates to the field of server system technology. More specifically, this application relates to a method and related products for testing server systems using automated scripts. Background Technology

[0002] The startup process of a server system is a complex and precise sequence from hardware power-on to the operating system being fully ready to provide services. The first stage of server system startup is the Power-On Self-Test (POST) stage. If a serious hardware or firmware failure is detected during the POST stage, the system will terminate the startup and issue an alert via beep or screen error code. Furthermore, if the POST stage hangs or enters an infinite loop, the entire system will fail to boot. The Fault Resilient Booting-2 (FRB-2) mechanism is a fault detection and recovery mechanism within the POST stage. This mechanism utilizes a watchdog timer in the server hardware to ensure that even if hardware or firmware failures occur during startup, the boot process can be completed or a manageable state can be entered to the greatest extent possible.

[0003] Server system testers need to test for various failures that may occur during the server system startup process to determine whether the server system will trigger the recovery procedure in FRB-2 under the circumstances of corresponding failures, in order to improve the reliability of the server system. However, the current FRB-2 mechanism testing relies on manual testing by testers. This testing mode is not only inefficient and prone to missing boundary scenarios, but also suffers from inaccurate test results due to its dependence on manual judgment.

[0004] In view of this, this application proposes a method and related products for testing server systems using automated scripts, in order to improve the efficiency and accuracy of FRB-2 mechanism testing. Summary of the Invention

[0005] In order to at least address one or more of the technical problems mentioned above, this application proposes methods, electronic devices, and solutions for non-transitory machine-readable storage media for testing server systems using automated scripts in several aspects.

[0006] In a first aspect, this application provides a method for testing a server system using automated scripts, comprising: injecting a simulated exception into the server system during the power-on self-test phase to trigger a server system crash; and determining whether the secondary fault recovery startup mechanism of the server system is normal based on the server system's recovery operation.

[0007] In a second aspect, this application provides an electronic device including: a processor; and a memory storing an automated script for testing a server system, wherein when the automated script is executed by the processor, the electronic device performs the method described in the first aspect.

[0008] In a third aspect, this application provides a non-transitory machine-readable storage medium having stored thereon an automated script for testing a server system, which, when executed by a processor, causes the method described in the first aspect to be implemented.

[0009] Using the methods and related products for testing server systems via automated scripts provided above, this application incorporates test code into the automated script. After the script runs, a simulated exception is injected during the POST phase to trigger a crash. The functionality of the FRB-2 mechanism is determined by monitoring the server system's recovery operations after the crash. In this way, the server system can automatically execute FRB-2 mechanism tests, reducing reliance on manual intervention and improving the efficiency and accuracy of testing. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein: Figure 1 Exemplary schematic diagrams of server systems employing the PCIe bus are shown in some embodiments of this application.

[0011] Figure 2 An exemplary swimlane flowchart of a power-on self-test method based on the FRB-2 mechanism in some embodiments of this application is shown. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be understood that the terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0015] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0016] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise expressly specified. "Several" means one or more, unless otherwise expressly specified.

[0017] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] Currently, information technology has permeated numerous industries, such as manufacturing, healthcare, finance, education, and agriculture, all of which are attempting to utilize information technology for intelligent transformation. With the development of information technology, all industries are seeking more efficient ways to process and store information, and server systems are one such option. Generally, a server system includes storage resources, processing resources, input / output devices, and a bus for connecting various hardware components. This bus can be implemented based on the Peripheral Component Interconnect Express (PCIe) standard.

[0019] Figure 1 Exemplary schematic diagrams of server systems employing the PCIe bus are shown in some embodiments of this application. For example... Figure 1As shown, in some embodiments, server system 100 includes: root complex (RC) 101, central processing unit (CPU) 102, main memory 103, switch 104, and endpoint device 105.

[0020] The root complex 101 is the starting point and root node of the PCIe hierarchy, directly connecting the CPU and main memory 103. Main memory 103 can be volatile memory used to provide the CPU 102 with the instructions and data required for computation in real time. Multiple CPUs 102 can be configured in the server system 100 to provide higher concurrent access and adapt to complex workloads.

[0021] continue Figure 1 The root complex 101, switch 104, and endpoint device 105 are all equipped with PCIe ports for establishing PCIe links. Switch 104 is used to expand PCIe connectivity, allowing one upstream port to be connected to multiple downstream ports. Endpoint device 105 is a leaf node in the PCIe hierarchy; in some embodiments, endpoint device 105 includes storage devices for storing data and instructions, such as high-speed solid-state drives or cache devices; in some embodiments, endpoint device 105 includes network devices for network connectivity, such as Ethernet network cards or wireless network cards; in some embodiments, endpoint device 105 includes input / output devices, such as a mouse, keyboard, or video display; in some embodiments, endpoint device 105 includes a power supply and power management devices for supplying power to various hardware components, distributing power, and controlling operating modes. Artificial intelligence (AI) technology has demonstrated a powerful influence and application value in fields such as natural language processing and computer vision. Neural network models are the core algorithms of AI technology. The development of AI technology has spawned large-scale models with hundreds of billions of parameters. The training and inference of these neural network models involve massive parallel computing and massive data throughput. In some embodiments, the endpoint device 105 includes an artificial intelligence processor to support AI application scenarios, such as Google's Tensor Processing Unit (TPU), Qualcomm's CloudAI 100 chip, and NVIDIA's A100, H100, and H200 graphics processing units (GPUs).

[0022] exist Figure 1In the server system 100, only a portion of the hardware components are shown. It will be apparent to those skilled in the art that the server system 100 may also include components related to… Figure 1 The hardware components shown are different from common hardware components. For example, the server system may include a motherboard, and the CPU 102, main memory 103, and some endpoint devices 105 may be integrated on the motherboard. In some embodiments, the motherboard integrates a Baseboard Management Controller (BMC) for monitoring the status of each hardware component and running hardware-level control programs. In addition, the server system 100 achieves flexible data manipulation through hardware and software collaboration. In some embodiments, the server system 100 can run operating systems such as Windows and Linux and implement various functions based on instruction sets. In some embodiments, the server system includes a Basic Input / Output System (BIOS) for providing a configuration interface and initializing the hardware components in the server system to bring them into a working state.

[0023] The startup process of a server system is a complex and precise sequence from hardware power-on to the operating system being fully ready to provide services. The first stage of server system startup is the Power-On Self-Test (POST) stage. Specifically, the POST is performed by the BIOS, and its operations include: checking the server system's critical hardware components, such as the CPU, main memory, network devices, and input / output devices, to ensure they are in normal working order; initializing the server system's critical hardware components and allocating system resources to them. If the POST stage detects a serious hardware or firmware failure (such as complete CPU or memory failure), the system will terminate the startup and issue an alert via a beep or screen error code. In addition, if the POST stage hangs or gets stuck in an infinite loop, the entire system will fail to boot.

[0024] Fault Resilient Booting (FRB) is a graded fault recovery mechanism for server systems, usually divided into multiple levels, the most common being FRB-1, FRB-2, and FRB-3. Each level corresponds to different fault scenarios and recovery measures. Among them, FRB-2 is a fault detection and recovery mechanism in the power-on self-test phase.

[0025] Figure 2 An exemplary swimlane flowchart of a power-on self-test method 200 based on the FRB-2 mechanism in some embodiments of this application is shown. Figure 2 As shown, in some embodiments, the execution entities of method 200 include the BIOS and the BMC, and the execution steps include: S201, The server system is powered on and the BIOS starts. In some embodiments, after the server system is powered on, the BIOS starts and executes the POST program.

[0026] S202, the BIOS establishes communication with the BMC. In some embodiments, after the BIOS starts up, it immediately establishes communication with the BMC and sends an FRB-2 enable request, while transmitting preset parameters, including the total duration of the timer and the action strategy in case of timeout.

[0027] S211, BMC activates watchdog timer. In some embodiments, BMC activates the internal watchdog timer (or FRB-2 timer) and sets the countdown according to the parameters passed by the BIOS. In some embodiments, the watchdog timer starts counting down once it is started, and the timer must be reset before it reaches zero; otherwise, BMC will consider the system to have lost responsiveness and crashed after the timer reaches zero, thereby triggering a preset boot or recovery action.

[0028] S203, the BIOS sequentially performs detection actions and periodically feeds the watchdog timer. In some embodiments, the BIOS sequentially detects the CPU, main memory, and input / output devices. After completing the detection of the previous object, it sends a feed signal to the BMC to reset the watchdog timer. In some embodiments, the BIOS sends a "feed" signal to the BMC between the execution of various key steps of POST to reset the watchdog timer. This action indicates that POST is still running normally and the system has not suspended.

[0029] S212, BMC resets watchdog timer. In some embodiments, BMC resets watchdog timer after receiving a feed signal.

[0030] In step S204, the BIOS transfers control to the operating system and sends an FRB-2 disable instruction to the BMC. In some embodiments, after completing all detection tasks, if no faults are detected, the BIOS confirms that the hardware is fault-free and the configuration is compliant, and then transfers control to the operating system bootloader; in these embodiments, the BIOS also sends an FRB-2 disable instruction to the BMC.

[0031] S214, BMC stops watchdog timer and disables FRB-2.

[0032] Understandably, if the BIOS gets stuck in an infinite loop or completely stops responding during the POST process due to hardware failure, firmware failure, or other reasons, it will be unable to reset the watchdog timer, and the watchdog timer will eventually reach zero. Therefore, method 200 also includes S213: if the watchdog timer countdown ends, the BMC determines that a POST failure has occurred. In some embodiments, the BMC determines that a serious failure has occurred in the server system during the POST phase when the timer times out, and executes a predefined recovery strategy. In some embodiments, the recovery strategy includes: the BMC recording a fault message in its System Event Log (SEL) for administrator diagnostics; and forcibly restarting the server system, thereby giving the server system a chance to retry completing the POST.

[0033] As understood from Method 200, the FRB-2 mechanism utilizes a watchdog timer in the server hardware to ensure the boot process can be completed or the system can enter a manageable state as much as possible even if hardware or firmware failures occur during startup. Many types of failures can occur during server system startup, including CPU-related failures, memory-related failures, motherboard core failures, and peripheral and expansion device failures. For server system testers, they need to test for various failures that may occur during startup to determine whether the server system will trigger the recovery procedure in FRB-2 under corresponding failures, thereby improving the reliability of the server system. However, current FRB-2 mechanism testing mainly relies on testers manually simulating server crashes. Testers need to record BMC logs and restart times, and verify the watchdog timer's timeout triggering and disabling logic. This testing mode is not only inefficient and prone to missing boundary scenarios, but also relies on manual judgment, leading to inaccurate test results.

[0034] In view of this, this application proposes a method and related products for testing server systems using automated scripts, in order to improve the efficiency and accuracy of FRB-2 mechanism testing.

[0035] In some embodiments, the method for testing a server system using automated scripts includes: injecting a simulated exception into the server system during the power-on self-test phase to trigger a server system crash; and determining whether the server system's secondary fault recovery startup mechanism is functioning correctly based on the server system's recovery operation.

[0036] The method proposed in this application tests the server system using automated scripts. Furthermore, this method can perform functional testing on the Fail-Recovery Startup-2 (FRB-2) mechanism. The automated test scripts used are executable code written according to preset logic. Running the automated test scripts guides the server system under test to complete the test steps. In some embodiments, test logs are generated at the end of the test for analysis by testers.

[0037] By introducing simulated exceptions during the POST phase, the server system can be crashed. In some embodiments, an automated script establishes a stable connection with the server system via the IPC-CLI protocol and crashes the server by invoking an error-injecting tool. In these embodiments, the automated script executes the error-injecting command halt() to force the server system to crash during the POST hardware test.

[0038] Under normal FRB-2 conditions, the server system will immediately perform recovery operations after a crash; conversely, if the FRB-2 mechanism malfunctions, the server system will be unable to perform recovery operations or will not be able to perform recovery operations in a timely manner. In some embodiments, if the server system performs recovery operations within a preset time range after a crash, the server system's secondary fault recovery startup mechanism is determined to be normal; if the server system does not perform recovery operations within the preset time range after a crash, the server system's secondary fault recovery startup mechanism is determined to be faulty.

[0039] Understandably, the method proposed in this application involves writing test code within an automated script. After the script runs, a simulated exception is injected during the POST phase to trigger a crash. The functionality of the FRB-2 mechanism is determined by monitoring the server system's recovery operations after the crash. In this way, the server system can automatically execute FRB-2 mechanism tests, reducing reliance on manual labor, lowering manpower costs, and reducing long-term verification costs, thereby improving the efficiency and accuracy of testing.

[0040] In some embodiments, a timer is configured during the power-on self-test phase to detect server system failures; in response to the timer resetting to zero, the server system: performs a recovery operation; and generates a record of the performed recovery operation in the logs associated with the secondary fault recovery startup mechanism.

[0041] In some embodiments, the timer can be implemented as the FRB-2 timer set in the BMC as described above, used to detect whether the server system has entered a crash state during the POST phase. Once the timer is started, it will begin counting down. The timer must be reset before it reaches zero; otherwise, the BMC will consider the system to have lost responsiveness and crashed after the timer reaches zero, thereby triggering a preset recovery action.

[0042] In some embodiments, if the timer returns to zero, assuming the server system's FRB-2 mechanism is functioning correctly, the server system will perform a recovery operation. Simultaneously, a record of the recovery operation will be generated in the logs related to the FRB-2 mechanism, including the start time of the recovery operation. In some embodiments, the recovery operation is performed by the BMC; in these embodiments, the BMC also records the start time of the recovery operation in its system event log.

[0043] In some embodiments, the server system is tested using multiple test cases; before testing using the current test case: the corresponding configuration items in the server system are set according to the requirements of the parameters related to the secondary fault recovery startup mechanism in the current test case.

[0044] In these embodiments, multiple test cases are used to test for various failures that may occur during the server system startup process, thereby achieving comprehensive coverage of all possible failure types during server system startup. Furthermore, by automatically traversing all parameter combinations, the coverage bottleneck of manual testing can be overcome, enabling automatic enumeration of all boundary scenarios and helping to avoid omissions that occur in manual testing.

[0045] To determine whether the server system will trigger the recovery procedure in FRB-2 in the event of a corresponding failure, so as to improve the reliability of the server system.

[0046] The operating logic of the FRB-2 mechanism is determined by relevant parameters, which can be configured through BIOS settings. Since testing requires multiple test cases, the parameters applied to different test cases may differ. In some embodiments, the test cases specify the requirements for FRB-2 mechanism parameters. Based on these requirements, the BIOS configuration items are set, and the server system is restarted after the settings are complete.

[0047] In some embodiments, the relevant parameters include at least one of the following: a secondary fault recovery startup mechanism switch, the total duration of the timer, and the recovery strategy.

[0048] The FRB-2 mechanism switch is a main switch used to control whether the FRB-2 mechanism is turned on or off, and its values ​​can include enable and disable.

[0049] In some embodiments, the total duration of the timer can be 3 to 30 minutes.

[0050] The recovery strategy refers to the strategy by which the system performs recovery operations after the FRB-2 timer expires. In some embodiments, the recovery strategy includes: performing no operation after the timeout; restarting the server system after the timeout; or cutting off the power supply to the server system after the timeout.

[0051] In some embodiments, the normality of the server system's secondary fault recovery startup mechanism is determined based on logs related to the secondary fault recovery startup mechanism.

[0052] In some embodiments, after performing the corresponding recovery operation according to the recovery strategy, the BMC records fault information and information about the recovery operation in its system event log. After the test is completed, the relevant records in the BMC's system event log can be used to determine whether the server system's FRB-2 mechanism is normal. For example, if information about the recovery operation is found in the record corresponding to the current test, it is determined that the server system's FRB-2 mechanism is normal; if no information about the recovery operation is found in the record corresponding to the current test, it is determined that the server system's FRB-2 mechanism is faulty.

[0053] In some embodiments, the recovery operation includes restarting the power-on self-test phase.

[0054] In some embodiments, the time difference between the first time when the simulated anomaly is injected into the server system and the second time when the server system performs a recovery operation is calculated; if the time difference is within a preset duration, it is determined that the secondary fault recovery startup mechanism of the server system is normal; if the time difference exceeds the preset duration, it is determined that the secondary fault recovery startup mechanism of the server system is faulty.

[0055] In these embodiments, the time required from a server system crash during the POST phase to the initiation of a recovery operation, assuming a normal FRB-2 mechanism, can be statistically analyzed, and this time can be used to set the value of the preset duration. If the time difference between the first and second times is within the preset duration, it can be assumed that the corresponding recovery operation was executed immediately after the crash during the POST phase, indicating that the server system's FRB-2 mechanism is normal. If the time difference between the first and second times exceeds the preset duration, it can be assumed that no recovery operation was executed after the crash during the POST phase, indicating a fault in the server system's FRB-2 mechanism.

[0056] It is understood that this application automatically records the first time of the server system injecting a simulated anomaly and the second time of the server system performing a recovery operation. The automatic timing error can be controlled within ±100ms, so the function of the FRB-2 mechanism can be accurately verified based on the time difference between the first and second times.

[0057] In some embodiments, the simulated anomaly includes at least one of the following: simulated CPU failure, simulated main memory failure, simulated network device failure, and simulated input / output device failure.

[0058] In summary, this application incorporates test code into automated scripts. Upon script execution, simulated exceptions are injected during the POST phase to trigger a crash. The functionality of the FRB-2 mechanism is determined by monitoring the server system's recovery operations after the crash. This allows the server system to automatically execute FRB-2 mechanism tests, reducing reliance on manual intervention and improving testing efficiency. Furthermore, multiple test cases are used to test various potential failures during server system startup, achieving comprehensive coverage of all possible failure types. Automated traversal of all boundary scenarios helps avoid omissions that occur during manual testing. Additionally, this application standardizes the testing process through scripts, eliminating human error and promoting standardization and consistency in testing. This ensures reproducible and comparable test results, and the automatic generation of structured reports facilitates problem backtracking and compliance auditing.

[0059] This application also discloses an electronic device, including: a processor; and a memory storing an automated script for testing a server system, wherein when the automated script is executed by the processor, the electronic device performs the method as described in any of the preceding embodiments.

[0060] This application also discloses a non-transitory machine-readable storage medium storing an automated script for testing a server system, which, when executed by a processor, enables the implementation of the methods described in any of the preceding embodiments.

[0061] The specific functions implemented by the electronic devices and non-transitory machine-readable storage media provided in the embodiments of this specification can be explained in comparison with the foregoing embodiments in this specification and can achieve the technical effects of the foregoing embodiments, so they will not be repeated here.

[0062] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for testing a server system using automated scripts, characterized in that, include: During the power-on self-test phase, a simulated anomaly is injected into the server system to trigger a server system crash. The server system's secondary fault recovery startup mechanism is determined to be functioning correctly based on the server system's recovery operation status.

2. The method according to claim 1, characterized in that, include: Configure a timer during the power-on self-test phase to monitor for server system crashes; In response to the timer returning to zero, the server system: Perform a recovery operation; Generate a record of the recovery operation in the logs related to the Level 2 fault recovery startup mechanism.

3. The method according to claim 2, characterized in that, include: The server system is tested using multiple test cases; Before testing with the current test case: Based on the requirements of the parameters related to the secondary fault recovery startup mechanism in the current test cases, the corresponding configuration items in the server system are set.

4. The method according to claim 3, characterized in that, The relevant parameters include at least one of the following: the secondary fault recovery startup mechanism switch, the total duration of the timer, and the recovery strategy.

5. The method according to claim 2, characterized in that, include: The server system's level 2 fault recovery startup mechanism is determined to be functioning correctly based on logs related to the level 2 fault recovery startup mechanism.

6. The method according to any one of claims 1-5, characterized in that, The recovery operation includes restarting the power-on self-test phase.

7. The method according to any one of claims 1-5, characterized in that, include: Calculate the time difference between the first time when the simulated anomaly is injected into the server system and the second time when the server system performs a recovery operation; In response to the time difference being within a preset time range, it is determined that the secondary fault recovery startup mechanism of the server system is normal; If the time difference exceeds the preset duration range, it is determined that the secondary fault recovery startup mechanism of the server system is faulty.

8. The method according to any one of claims 1-5, characterized in that, The simulated anomaly includes at least one of the following: simulated CPU failure, simulated main memory failure, simulated network device failure, and simulated input / output device failure.

9. An electronic device, characterized in that, include: processor; as well as A memory storing automated scripts for testing a server system, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-8.

10. A non-transitory machine-readable storage medium storing an automated script for testing a server system, wherein when the automated script is executed by a processor, the method of any one of claims 1-8 is implemented.

Images