Test method and electronic device

Abstract

The application discloses a test method and electronic equipment, and relates to the technical field of whole cabinet testing. An out-of-band test host is additionally arranged. The out-of-band test host establishes an out-of-band management channel with each to-be-tested node in a to-be-tested whole cabinet through a top switch. A binding relationship between the to-be-tested whole cabinet and each to-be-tested node is established based on the out-of-band test host. Subsequently, the to-be-tested nodes in the to-be-tested whole cabinet are subjected to out-of-band test operation according to the binding relationship through the established out-of-band management channel, so as to obtain hardware state information fed back by the to-be-tested nodes through the out-of-band management channel, and then the out-of-band test result of the to-be-tested whole cabinet is determined. Therefore, the technical problem that in-band test must wait for all computing nodes in the whole cabinet to complete system starting and driver loading before being executed, and is easily disturbed by test network and test environment factors to cause test failure can be solved, and the technical effects of improving whole cabinet test efficiency and test result accuracy are achieved.

Description

Technical Field

[0001] This application relates to the field of whole-rack testing technology, and in particular to a testing method and electronic equipment. Background Technology

[0002] With the rapid development of large-scale AI model training and high-performance computing, GPU racks, as high-density, high-performance integrated computing units, have become core infrastructure of data centers and intelligent computing centers. The hardware reliability and interconnection stability of the rack directly determine the continuity and efficiency of computing power output. In related technologies, in-band testing is commonly used to test the entire rack. However, in-band testing requires all computing nodes within the rack to complete system startup and driver loading before execution, and it is easily affected by factors such as the test network and test environment, leading to test failures. Summary of the Invention

[0003] This application provides a testing method and electronic device to at least solve the problem in related technologies that in-band testing can only be executed after all computing nodes in the rack have completed system startup and driver loading, and is easily affected by factors such as the test network and test environment, which can lead to test failure.

[0004] This application provides a testing method applied to an out-of-band test host. The out-of-band test host is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test within the rack under test through the top switch. The method includes: Obtain the identification information of the rack under test and the identification information of each node under test in the rack under test; Based on the identification information of the rack under test and the identification information of each node under test in the rack under test, establish the binding relationship between the rack under test and each node under test; Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, out-of-band testing operations are performed on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel; Based on the hardware status data fed back by each node under test through the out-of-band management channel, the out-of-band test results of the entire rack under test are determined.

[0005] This application also provides a testing device applied to an out-of-band test host. The out-of-band test host is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test within the rack under test through the top switch. The device includes: The acquisition module is used to acquire the identification information of the rack under test and the identification information of each node under test in the rack under test; The binding module is used to establish a binding relationship between the rack under test and each node under test based on the identification information of the rack under test and the identification information of each node under test in the rack under test. The testing module is used to perform out-of-band testing on each node under test based on the out-of-band management channel and the binding relationship between the rack under test and each node under test, so as to obtain the hardware status information fed back by the node under test through the out-of-band management channel. The result determination module is used to determine the out-of-band test results of the entire rack under test based on the hardware status data fed back by each node under test through the out-of-band management channel.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described test methods when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described test methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described test methods.

[0009] This application addresses the technical problem of adding an out-of-band test host. This out-of-band test host establishes an out-of-band management channel with each node under test (DUT) within the rack under test via a top switch. Based on the out-of-band test host, a binding relationship is established between the rack under test and each DUT. Subsequently, out-of-band testing is performed on the DUTs within the rack under test according to this binding relationship through the established out-of-band management channel. This allows for the acquisition of hardware status information fed back by the DUTs through the out-of-band management channel, thereby determining the out-of-band test results for the rack under test. Therefore, this solves the technical problem that in-band testing must wait for all computing nodes within the rack to complete system startup and driver loading before execution, and is easily affected by interference from factors such as the test network and test environment, leading to test failures. This achieves the technical effect of improving the testing efficiency and accuracy of rack under test results. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the test system on which the embodiments of this application are based; Figure 2 A flowchart illustrating the testing method provided in the embodiments of this application; Figure 3 A schematic diagram illustrating the consistency verification process of the identity information of the node under test provided in this embodiment of the application; Figure 4 A flowchart illustrating an exemplary testing method provided in an embodiment of this application; Figure 5 A schematic diagram of the target power supply disturbance operation provided in the embodiments of this application; Figure 6 A flowchart illustrating another exemplary testing method provided in this application embodiment; Figure 7 A flowchart illustrating another exemplary testing method provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the testing device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] With the rapid development of scenarios such as large-scale artificial intelligence (AI) model training and high-performance computing (HPC), GPU racks, as high-density, high-power integrated computing units, have become the core infrastructure of data centers and intelligent computing centers. A GPU rack integrates multiple Compute Trays, Switch Trays, high-speed backplanes, redundant power supplies, and cooling systems. Through the NVSwitch chip, it constructs an all-to-all GPU topology, enabling collaborative computing of hundreds or thousands of GPUs to support the training and inference tasks of trillion-parameter large-scale models.

[0016] The hardware reliability and interconnection stability of the GPU rack directly determine the continuity and efficiency of AI computing power output. Therefore, testing and verification of the GPU rack are crucial throughout its entire lifecycle, including factory testing, data center deployment testing, daily operation and maintenance, and fault location.

[0017] In related technologies, it is usually implemented in-band, with the test program running under the CT node OS. The entire test depends on the device operating system (OS), GPU driver, CUDA and other related business environments.

[0018] However, current in-band testing has the following problems: The test startup overhead is large and the parallel efficiency of the whole rack is low: In-band testing must complete the power-on self-test, BIOS initialization, OS kernel loading, driver mounting and service startup process on a node-by-node basis. The overall test pre-processing time is long and it is impossible to achieve second-level rapid inspection.

[0019] Batch testing involves high network and management pressure, which can easily lead to concurrency bottlenecks: When multiple nodes are deployed concurrently in a rack, the PXE network bandwidth, storage IO bottlenecks, and node startup timing conflicts can cause multiple nodes to simultaneously pull OS images, transmit test logs, and execute remote control commands, which can consume a large amount of in-band business network bandwidth, easily causing network congestion and packet loss, resulting in decreased stability and execution timeouts in batch concurrent testing.

[0020] The testing environment is highly dependent on various factors, and environmental anomalies can easily lead to test failures: Traditional in-band testing relies heavily on local in-band resources such as the PXE networking environment, operating system image, and hardware adaptation drivers. The environment has many components and high coupling. Testing requires a complete in-band runtime stack; any network anomaly, corrupted image file, or driver compatibility failure in any stage will directly cause test cases to fail to schedule properly, business processes to malfunction, leading to test failures, distorted results, weak environment fault tolerance, and poor overall test stability and reusability.

[0021] The test case execution chain is long and the fault location is complex: In-band testing covers the entire stack of hardware, BIOS, drivers, OS and application. Faults may occur at any level. When an anomaly occurs, it is necessary to check hardware adaptation, kernel parameters, driver conflicts, software logic and other issues layer by layer. Compared with out-of-band coarse-grained detection, the location cycle is significantly extended.

[0022] Inability to cover non-operational hardware scenarios such as power-off / power-on / off: In-band testing can only be carried out in OS runtime, and cannot verify the hardware reliability of non-OS runtime scenarios such as node power-off, hibernation, restart transition state, and remote power-on anomalies. It also has insufficient coverage of early hardware faults such as power timing and power-on initialization.

[0023] To address the aforementioned technical problems, this application provides a testing method and electronic device. By adding an out-of-band test host, which establishes an out-of-band management channel with each node under test within the rack under test via a top switch, a binding relationship is established between the rack under test and each node under test based on the out-of-band test host. Subsequently, out-of-band testing operations are performed on the nodes under test within the rack under test according to this binding relationship through the established out-of-band management channel. This obtains the hardware status information fed back by the nodes under test through the out-of-band management channel, thereby determining the out-of-band test results of the rack under test. Therefore, this solves the technical problem that in-band testing must wait for all computing nodes within the rack to complete system startup and driver loading before execution, and is easily affected by interference from factors such as the test network and test environment, leading to test failures. This achieves the technical effect of improving the testing efficiency and accuracy of rack under test results.

[0024] The specific application environment architecture or specific hardware architecture on which the execution of the test method depends is described here.

[0025] First, the structure of the testing system on which this application is based will be described: The testing method and electronic equipment provided in this application are suitable for out-of-band testing of a complete cabinet containing multiple nodes under test. Figure 1 The diagram shown is a structural schematic of the test system based on the embodiments of this application, mainly including a rack under test and an out-of-band test host. The out-of-band test host is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test within the rack under test through the top switch. The out-of-band test host is used to perform out-of-band testing on multiple nodes under test within the rack under test.

[0026] This application provides a testing method for out-of-band testing of a server rack containing multiple nodes under test. The execution subject of this application embodiment is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic device that can serve as the out-of-band test host for the server rack under test.

[0027] like Figure 2 The diagram shown is a flowchart of a testing method provided in an embodiment of this application. The method includes: Step 201: Obtain the identification information of the rack under test and the identification information of each node under test in the rack under test.

[0028] It should be noted that the testing method provided in this application embodiment is applied to an out-of-band test host. The out-of-band test host is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test in the rack under test through the top switch.

[0029] Among them, the identification information is a unique identifier set at the factory for the entire rack under test and the hardware node under test.

[0030] Step 202: Based on the identification information of the rack under test and the identification information of each node under test in the rack under test, establish the binding relationship between the rack under test and each node under test.

[0031] It should be noted that in the in-band testing scenario of the entire rack, after the node starts the operating system, it will actively report its own identity information and the identity information of the rack to which it belongs. However, the embodiment of this application provides an out-of-band testing method. Out-of-band testing can be performed when the node has not started the operating system. The node cannot actively report its own identity information and the identity information of the rack to which it belongs through the software layer. Therefore, it is necessary to obtain the identity information of the rack to be tested and the identity information of each node to be tested in the rack to be tested before testing, and then establish the binding relationship between the rack to be tested and each node to be tested.

[0032] Out-of-band testing, as a testing method that does not rely on the device's operating system (OS), drivers, and business environment, interacts directly with the device hardware through out-of-band management channels (such as BMC, IPMI, and Redfish). It can be used to conduct tests in scenarios such as bare metal and cold start, avoiding interference from the OS, drivers, and business load on the test results.

[0033] Step 203: Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, perform out-of-band testing operations on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel.

[0034] It should be noted that the out-of-band management channel, also known as the hardware management channel, is a channel that is independent of the service network of the node under test, based on the baseboard management controller (BMC) built into the node under test. It can realize hardware status acquisition and control without loading the node's operating system and service drivers.

[0035] Specifically, the out-of-band test host, through a pre-established out-of-band management channel and according to the established binding relationship, issues corresponding out-of-band test operation commands to the corresponding nodes under test. It can directly collect hardware status data from the nodes under test via the BMC (Hardware Management Channel) without waiting for the internal operating system to start and drivers to load. Furthermore, the out-of-band test operations on the nodes under test can be performed in parallel across nodes, improving the overall rack testing efficiency.

[0036] The out-of-band test host, having pre-established a binding relationship between the rack under test and each node under test, can accurately send out-of-band test commands to the corresponding nodes under test based on this binding relationship when issuing out-of-band test operation commands. Hardware status data may include the voltage, frequency, power consumption, temperature, and link connectivity status of the internal devices of the nodes, which can be determined according to the type of out-of-band test operation being performed.

[0037] Step 204: Determine the out-of-band test results of the entire rack under test based on the hardware status data fed back by each node under test through the out-of-band management channel.

[0038] Specifically, the out-of-band test host aggregates the hardware status information fed back by all nodes under test and performs an overall analysis of all data. If the hardware status of all nodes meets the preset qualification standards, the out-of-band test of the entire rack is deemed to have passed; if the hardware status of any node does not meet the qualification standards, the out-of-band test of the entire rack is deemed to have failed, and a corresponding out-of-band test report is generated.

[0039] Based on the above embodiments, as an implementable approach, in one embodiment, a binding relationship is established between the rack under test and each node under test according to the identification information of the rack under test and the identification information of each node under test in the rack under test, including: Step 2021: For any node under test in the rack under test, perform a consistency check on the identity information of the node under test. Step 2022: If the identity information of the node under test passes the consistency verification, establish a binding relationship between the rack under test and the node under test based on the identity information of the rack under test and the node under test.

[0040] It should be noted that since this application embodiment implements an out-of-band test, the identification information of the rack under test and the identification information of each node under test in the rack under test are generally manually entered by the operator. If the manual entry method using a barcode scanner is used, there may be entry errors. Therefore, before establishing the binding relationship, this application first performs consistency verification on the identification information of the node under test to avoid subsequent binding errors due to information collection errors, which would affect the accuracy of the test results of the rack under test.

[0041] Specifically, in one embodiment, the out-of-band test host is equipped with a Dynamic Host Configuration Protocol (DHCP) environment. For any node under test (DUT) in the rack under test, it can search for the target physical address of the DUT from a preset configuration database based on the DUT's identity sequence code. The DUT's identity information includes at least an identity sequence code. Based on the DHCP environment, a target logical address is determined for the target physical address. The DUT's identity sequence verification code is obtained based on the target logical address. If the DUT's identity sequence verification code and identity sequence code match, the DUT's link layer discovery protocol neighbor information is obtained. Based on the link layer discovery protocol neighbor information, the DUT's target physical verification address is determined. If the DUT's target physical verification address matches the target physical address, the DUT's identity information is determined to have passed the consistency check.

[0042] Accordingly, in one embodiment, if the identity sequence verification code and the identity sequence code of the node under test are inconsistent, or if the target physical verification address and the target physical address of the node under test are consistent, it is determined that the node under test has a consistency verification anomaly, so as to trigger the process of re-acquiring the identity information of the node under test.

[0043] It should be noted that the out-of-band test environment must be set up before starting the test process. The out-of-band test host provided in this embodiment is a stable and reliable micro-host with a Linux system installed, Secure Shell (SSH) service enabled to support remote login, and a Dynamic Host Configuration Protocol (DHCP) environment set up to automatically assign Internet Protocol (IP) addresses to the management ports of each node in the rack.

[0044] In terms of physical connections, the test host is connected to the aggregation switch (the top switch of the rack under test) via an external Universal Serial Bus Optical Network Interface Card (USB optical NIC). The four out-of-band (OOB) switches at the top of the rack are connected in series via network cables to form a stacked network. The eight power shelves below the rack are each connected to the OOB switch via eight network cables. The internal network cables connect the management ports of the compute tray (CT), switch tray (ST), BlueField-3 data processing unit (BF3), and coolant distribution unit (CDU) to the OOB switch. At this point, the out-of-band test host, together with the CT, ST, Powershelf, BF3, CDU, and OOB switches within the rack under test, constitute a hardware management LAN independent of the service network. Testers can use their laptops to connect to the out-of-band test host and issue commands to each node under test within the rack through the out-of-band test host. This out-of-band test environment deployment achieves quick, convenient, and efficient network configuration, and can flexibly handle various delivery test scenarios.

[0045] Specifically, such as Figure 3The diagram illustrates the consistency verification process for the identity information of the node under test provided in this embodiment. For any node under test in the rack under test, the out-of-band test host retrieves the manually entered identity serial number (SN) of the node under test, searches for an entry matching the identity serial number in the preset configuration database (order information database), and obtains the target physical address of the node. The out-of-band test host calls the lease record of the locally built Dynamic Host Configuration Protocol (DHCP) service to find the target logical address automatically assigned to the target physical address (BMC MAC address). The out-of-band test host sends a hardware information query command to the target logical address (BMCIP) through the Intelligent Platform Management Interface (IPMI) protocol: ipmitool -I lanplus -H $IP -U admin -P admin fru list0, and reads the hardware serial number embedded in the BMC Field Replaceable Unit (FRU) information as the identity serial number verification code of the node.

[0046] Furthermore, if the identity sequence check code (SN1) matches the identity sequence code (SN), the out-of-band test host logs into the top switch and executes the command: `show lldp neighbors detail` to obtain the Link Layer Discovery Protocol (LLDP) neighbor information. This information includes data such as the MAC addresses of neighboring devices connected to each switch port. The LLDP neighbor information corresponding to the current node under test is extracted from this information to obtain the target physical verification address of that node. The target physical verification address is compared with the target physical address. If they match, the identity information of the node under test is determined to have passed the consistency check; if they do not match, the check fails, and the identity information re-acquisition process is triggered.

[0047] Based on the above embodiments, Figure 4 This is a flowchart illustrating an exemplary testing method provided in an embodiment of this application. As an implementable approach, in one embodiment, based on an out-of-band management channel, out-of-band testing is performed on each node under test according to the binding relationship between the rack under test and each node under test, to obtain hardware status information fed back by the node under test through the out-of-band management channel, including: Step 2031: Detect the target test time window for the entire rack under test; Step 2032: Within the target test time window, based on the out-of-band management channel and according to the binding relationship between the rack under test and each node under test, perform target processor performance test operations on each node under test to obtain the target processor status information fed back by the node under test through the out-of-band management channel. The out-of-band test operations include target processor performance test operations, which include AC power reset operations, and the hardware status information includes target processor status information.

[0048] It should be noted that target processor performance testing is one of the test items in computing unit testing. The target processor can be the Central Processing Unit (CPU). CPU latent performance defects are characterized by strong concealment and difficulty in detection by static testing. Conventional static data acquisition can only capture the performance of the CPU under normal operating conditions and cannot trigger the manifestation of latent defects. CPU performance defects can be intercepted by active fault triggering. Specifically, active fault triggering can be achieved through AC power reset operation.

[0049] It should be further explained that the target test time window is the unbalanced transition phase during the process of the system recovering to steady state after the CPU power supply disturbance ends. Performing an AC reset operation during this phase can maximize the CPU defect trigger rate.

[0050] Specifically, after determining the target test time window for the entire rack under test, an AC reset command can be sent to the node under test to control the node to power on and off (AC reset), and the target processor status information after the AC reset can be obtained from the node under test. The target processor status information includes CPU performance indicators and abnormal behavior data, thereby obtaining the activation status of latent faults and the system stability performance.

[0051] Specifically, in one embodiment, a target power supply disturbance operation can be performed on the rack under test to obtain the defect trigger rate of the rack under test in multiple time windows; the time window with the highest defect trigger rate is taken as the time window to be corrected; and the target test time window of the rack under test is determined according to the time window to be corrected and the target operation parameters corresponding to the target power supply disturbance operation.

[0052] Specifically, the out-of-band test host can perform target power supply disturbance operations on the entire rack under test according to a tiered power supply disturbance strategy to induce CPU performance defects. The experiment can be designed using a control experiment and a multi-window scanning method. First, the power supply disturbance parameters (disturbance amplitude) are set, and the AC trigger time is defined. , Indicates the AC reset trigger time. Indicates the end time of the disturbance. Also known as AC reset trigger offset, it divides the recovery process after the disturbance ends into 4 time intervals: Time window 1: Δt∈[0,0.2τ] (early triggering of the recovery phase); Time window 2: Δt∈[0.2τ,0.6τ] (triggered in the middle of the recovery phase); Time window 3: Δt∈[0.6τ,τ] (triggered in the later stage of the recovery phase); Time window 4: Δt > τ (approaching no disturbance).

[0053] The Powershelf was used to induce a disturbance for a duration of T2, either by power-off or current-limiting. AC reset was triggered according to different Δt intervals. Data was collected for each interval, yielding the following experimental results: Defect trigger rate 5% in time window 1, 35% in time window 2, 10% in time window 3, 3% in time window 4, and 1% in the absence of disturbance. Based on these experimental results, the defect trigger rate can be determined. The maximum value is obtained in the interval (0.2, 0.6), that is, time window 2 is the time window to be corrected, and then the corresponding target test time window is determined according to time window 2.

[0054] Specifically, in one embodiment, a first-level power supply disturbance operation can be performed on the cabinet under test to obtain the first-level power supply disturbance operation result; a second-level power supply disturbance operation can be performed on the cabinet under test to obtain the second-level power supply disturbance operation result; and the defect triggering rate of the cabinet under test in multiple time windows can be determined by combining the first-level power supply disturbance operation result and the second-level power supply disturbance operation result.

[0055] The first-level power supply disturbance operation is used to cause a global power supply drop in the cabinet under test, while the second-level power supply disturbance operation is used to cause a local power supply imbalance in the cabinet under test.

[0056] Specifically, this application employs a graded power supply disturbance and multi-stage judgment method. By applying power supply disturbances of different granularities at each stage and performing AC reset and status judgment during the disturbance recovery process, the method achieves the gradual activation and location of latent faults. The graded disturbance involves two levels of power supply disturbance. The first level is a cabinet-level disturbance, achieved by shutting down four powershelves, resulting in a global power supply drop. The second level is a power supply unit (PSU) level disturbance, achieved by shutting down the first PSU in each powershelf, introducing local imbalance disturbances. This causes the system to form a spatially and temporally superimposed unbalanced state during the recovery phase, exhibiting both residual disturbance and preliminary stability characteristics. Through comprehensive experiments with these two levels of disturbances, both global and local power supply anomaly scenarios are covered. Cross-validation ensures the reliability of the defect trigger rate statistics, avoiding the high randomness of experimental results and laying the foundation for improving the accuracy of subsequent target test time window determination.

[0057] Furthermore, by analyzing the results of the first-level power supply disturbance operation and the second-level power supply disturbance operation, the overall variation law of the defect occurrence rate under different power supply disturbances is determined, and the defect triggering rate of the cabinet under test in multiple time windows is obtained.

[0058] Specifically, in one embodiment, the target test time window for the entire rack under test can be determined based on the following formula:

[0059] in, This indicates the target test time window for the entire rack under test. Indicates the time window to be corrected. This indicates the power supply disturbance recovery speed of the entire cabinet under test, also known as the recovery time constant. This represents the optimal scaling factor. ,Right now , This indicates the preset correction factor. This indicates the disturbance magnitude; the target operating parameters (power supply disturbance parameters) include the disturbance magnitude.

[0060] Specifically, through statistical analysis and comparison of experimental data, the relationship between fault induction capability and system stability under different recovery stages is extracted. It is determined that within this interval, the system is in a critical state of instability but recovery. Furthermore, the discrete experimental results are expressed as a function. Finally, the AC reset trigger time (target test time window) is uniformly expressed as a function of the disturbance duration and the recovery time constant τ, with the disturbance amplitude A introduced as a correction term. This achieves a mapping from experimental data to a function defining the target test time window. The target test time window corresponds to the non-equilibrium transition stage in the system's recovery from disturbance to steady state. Performing AC reset during this stage can significantly increase the probability of latent defect activation.

[0061] For example, such as Figure 5 The diagram illustrates the process of the target power supply disturbance operation provided in this embodiment. First, the node under test is powered on to ensure that the BIOS has booted and completed. The BMC log "System Boot Initiated BIOS_Boot_Up" is captured out-of-band. If this log exists, it indicates that the BIOS boot of the node under test has completed and the system is stable; otherwise, the process continues to wait. CPU system bus power supply, CPU power consumption, and real-time CPU frequency performance indicators are collected, along with boot status and error logs, to determine faults. CPU power consumption, frequency, voltage, and power supply logs are collected using the Redfish interface (out-of-band management channels include the Redfish interface). The normal voltage threshold range is 0.4V-1.3V, the power consumption threshold range is greater than 40W, and the motherboard frequency is greater than 1.8GHz. Values ​​outside this threshold range are considered to indicate a CPU performance problem. If no CPU performance problem is confirmed, a first-level power supply disturbance is applied. The eight Powershelves are powered in a 4-main, 4-backup configuration. Any four Powershelves are shut down to create a global power supply voltage fluctuation. Once the AC trigger condition is met, AC is triggered, and data is collected and analyzed. After data acquisition and evaluation, a second-level power supply disturbance is applied. The first PSU of the eight Powershelf units is shut down, while the remaining five PSUs are used to generate power, creating a localized power supply voltage disturbance. Once the AC trigger condition is met, AC is triggered, and data acquisition and evaluation are resumed. The out-of-band test host can remotely control the motherboard's power-on and power-off via the ipmitool command to perform an AC power-off-reboot operation on the server under test, clearing the CPU hardware cache, resetting the BIOS power consumption policy and VRM power supply status, simulating a node restart in a real-world scenario.

[0062] Based on the above embodiments, Figure 6The flowchart illustrates another exemplary testing method provided in this application. As an implementable approach, in one embodiment, based on an out-of-band management channel, out-of-band testing is performed on each node under test according to the binding relationship between the rack under test and each node under test, to obtain hardware status information fed back by the node under test through the out-of-band management channel, including: Step 2033: Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, send the first fan control command to each test node to control the fan of the test node to rotate at the first speed to generate first-level mechanical vibration, and obtain the first leakage detection circuit signal fed back by the test node through the out-of-band management channel. Step 2034: Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, send a second fan control command to each test node to control the fan of the test node to rotate at a second speed to generate secondary mechanical vibration, and obtain the second leakage detection loop signal fed back by the test node through the out-of-band management channel.

[0063] The second rotational speed is greater than the first rotational speed. The hardware status information includes the first leakage detection circuit signal and the second leakage detection circuit signal. The first leakage detection circuit signal and the second leakage detection circuit signal are used to characterize the health status of the leakage detection line.

[0064] It should be noted that the leakage detection line is a test item in the computing unit test. The leakage sensing cable and detection cable are arranged inside the chassis, in the gaps between boards, and around the liquid cooling pipeline. The wiring space is small and the layout is compact. During the production and assembly process, problems such as loose cable connections, incomplete terminal insertion, unlocked clips, and suspended or missing wires are very likely to occur. These defects have no obvious external damage and do not report errors immediately. Conventional visual inspection and single-point continuity testing cannot effectively identify them, and they are typical hidden assembly failures.

[0065] Specifically, leakage detection data can be collected first. If the current leakage detection data does not reflect any abnormalities, further dynamic disturbances can be applied to the leakage detection cable by controlling the fan speed under different operating conditions to generate mechanical vibrations. Combined with out-of-band continuous signal acquisition and fluctuation judgment, hidden assembly defects such as loose connections and poor contact can be identified. Two levels of mechanical vibration disturbances are applied to the computing node (the node under test), generating mechanical vibrations of different intensities and global resonance with the chassis. The continuous vibration from the fan amplifies the minute displacement of the wiring harness, causing the loose connection terminal to momentarily disconnect and the signal to jump, making the originally hidden poor contact problem explicit. The on / off signal and alarm status of the leakage detection circuit are continuously collected out-of-band through the BMC (out-of-band management channel). The first leakage detection circuit signal and the second leakage detection circuit signal are used to characterize the health of the leakage detection line. If the wiring harness is loose, there will be signal discontinuity, level jumps, and intermittent alarms. For qualified prototypes with intact wiring and no loose connections, the detection signal is continuously stable and without fluctuations during vibration, which can accurately distinguish between qualified and defective parts without manual re-inspection.

[0066] Furthermore, after acquiring the signals from the first and second leakage detection circuits, the automatic speed control of the fan is restored to ensure normal operation of the fan.

[0067] Based on the above embodiments, Figure 7 The flowchart illustrates another exemplary testing method provided in this application. As an implementable approach, in one embodiment, based on an out-of-band management channel, out-of-band testing is performed on each node under test according to the binding relationship between the rack under test and each node under test, to obtain hardware status information fed back by the node under test through the out-of-band management channel, including: Step 2035: Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, perform a global reset operation on each node under test to trigger the rack under test to retrain all target links and obtain the target link topology-level verification information fed back by the node under test through the out-of-band management channel.

[0068] The hardware status information includes target link topology-level verification information.

[0069] It should be noted that the target link can be an NVLink link. An NVLink link is the core high-speed data transmission link between the GPU and CPU, and between GPUs in the rack under test. It covers the entire process, including chip interfaces, NVLink cables, connectors, link topology, and protocol interaction. The integrity and transmission stability of this link directly determine the overall computing power output efficiency of the rack. This embodiment of the application detects NVLink links by performing tiered power disturbances. By applying global and local power reset operations (global reset operation) to the nodes under test in the rack, link retraining is triggered. Combined with multi-node link status information, the detection and location of latent faults in the NVLink link are achieved.

[0070] Specifically, the external test host, based on the established binding relationship, sends a global AC power reset command to the nodes under test (DUTs) within the rack via the BMC out-of-band management channel to control the power-off restart of all DUTs. Since the NVLink link status is only established during the link training phase of node power-on initialization, the global reset operation triggers all DUT GPUs to re-enter the initialization process, thereby triggering a full retraining of all NVLink links within the rack. This full retraining includes physical layer signal calibration, link rate negotiation, and connectivity verification. After the link retraining is complete, the test host, through the Redfish interface (out-of-band management channel), collects the NVLink link training results (target link topology-level verification information) fed back by each DUT in parallel. This information directly reflects the topology health of the entire rack's NVLink links.

[0071] Specifically, by triggering full link retraining through a global reset, all NVLink links within the rack under test are tested in one go. This solves the problem that traditional single-node testing cannot cover the entire link across nodes and is difficult to detect topology errors. Since NVLink links are established during the startup / training phase, triggering link retraining through a global reset of the node under test and renegotiation of the protocol layer through power events exposes hidden problems, thereby improving the accuracy of out-of-band test results for the entire rack.

[0072] Based on the above embodiments, as an implementable approach, in one embodiment, the node under test includes a compute node under test and a switching node under test. The out-of-band test results of the entire rack under test are determined based on the hardware status data fed back by each node under test through the out-of-band management channel, including: Step 2041: Based on the first link status information fed back by each computing node under test through the out-of-band management channel and the second link status information fed back by the switching node under test through the out-of-band management channel, perform bidirectional verification on the first link status information and the second link status information corresponding to the same target link in the rack under test, and obtain the bidirectional verification result. Step 2042: Determine the test results of the target links of the rack under test based on the bidirectional verification results of each target link within the rack under test.

[0073] The target link topology-level verification information includes the first link status information and the second link status information, and the out-of-band test results include the target link test results.

[0074] It should be noted that bidirectional verification is a method of cross-verifying link status data from both ends of the same target link, on the CT side (the compute node under test) and the ST side (the switching node under test). This method offers higher accuracy compared to single-sided link status data verification. This application's embodiment, based on out-of-band BMC control, Redfish interface (out-of-band management channel) acquisition, and multi-condition linkage strategies, systematically solves the problem of out-of-band test coverage across the entire NVLink link, achieving non-intrusive testing of the entire NVLink link, all stages, and all scenarios within the entire rack. The second link status information includes platform environment, service status, NVLink negotiation results (link training results), and NVLink interface status information.

[0075] Specifically, in one embodiment, for any target link, if the first link state information of the target link indicates that the link training is successful and the second link state information indicates that the link training is successful, the bidirectional verification result of the target link can be determined to be normal; if the first link state information of the target link indicates that the link training is unsuccessful or the second link state information indicates that the link training is unsuccessful, the bidirectional verification result of the target link can be determined to be abnormal.

[0076] It should be noted that NVLink is a cross-device link. Successful training on the compute node under test only indicates that the port on the compute node under test is normal, and does not reflect the NVSwitch port status of the switch node under test; conversely, successful training on the switch node under test does not reflect that the port on the compute node under test is normal.

[0077] Specifically, the bidirectional verification result of the target link is determined to be normal only if both the first link state information and the second link state information indicate that the link training was successful; otherwise, the bidirectional verification result of the target link is determined to be abnormal.

[0078] Specifically, such as Figure 7As shown, this embodiment of the application provides more accurate fault location through node-by-node perturbation. First, NVLink link training results are collected in the initial state, and an initial consistency check (bidirectional check) is performed to filter errors in the training phase (first round of judgment) to determine the bidirectional check results of each target link. Then, a global AC reset is performed on all nodes under test (CT nodes), and the NVLink link training results are acquired again for a second round of judgment to redetermine the bidirectional check results of each target link. Then, node-by-node perturbation (AC reset) is performed on each node under test to obtain multiple rounds of judgment results, and the bidirectional check results of each target link are repeatedly determined, thereby accurately locating the implicit link anomaly caused by a single node failure and ensuring the stability of the entire link topology.

[0079] Accordingly, in one embodiment, if the bidirectional verification result of any target link is abnormal, the test computing node corresponding to the first link status information and the test switching node port corresponding to the second link status information are marked as abnormal points.

[0080] Specifically, the first link status information that reports anomalies can be filtered to identify the corresponding computing node under test. At the same time, the second link status information that reports anomalies can be filtered to identify the corresponding switching node port under test. Then, the computing node under test and the switching node port under test can be marked as anomalies for subsequent troubleshooting and maintenance.

[0081] Specifically, in one embodiment, if the bidirectional verification results of all target links in the rack under test are normal, the target link test result of the rack under test is determined to be normal; if the bidirectional verification result of any target link in the rack under test is abnormal, the target link test result of the rack under test is determined to be abnormal.

[0082] Specifically, if the bidirectional verification result of any target link is abnormal, that is, at least one target link training has failed, the target link test result of the entire cabinet under test is directly determined to be abnormal, indicating that there is a break or instability in the NVLink topology of the entire cabinet, thus improving the reliability of the target link test result.

[0083] The testing method provided in this application, by adding an out-of-band test host, establishes an out-of-band management channel between the out-of-band test host and each node under test within the rack under test via a top switch. Based on the out-of-band test host, a binding relationship is established between the rack under test and each node under test. Subsequently, out-of-band testing operations are performed on the nodes under test within the rack under test according to this binding relationship through the established out-of-band management channel to obtain the hardware status information fed back by the nodes under test through the out-of-band management channel, thereby determining the out-of-band test results of the rack under test. Therefore, it can solve the technical problem that in-band testing must wait for all computing nodes within the rack to complete system startup and driver loading before execution, and is easily affected by factors such as the test network and test environment, leading to test failures. This achieves the technical effect of improving the testing efficiency and accuracy of rack under test results. Furthermore, it overcomes the limitation of traditional static testing in failing to detect latent defects by introducing disturbances, hierarchical strategies, and AC reset, thus improving the latent fault detection capability.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0085] The embodiments of this application also provide a testing device for performing the testing method provided in the above embodiments. The device is applied to an out-of-band test host, which is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test in the rack under test through the top switch.

[0086] like Figure 8 The diagram shown is a structural schematic of the testing device provided in an embodiment of this application. The testing device 80 includes: an acquisition module 801, a binding module 802, a testing module 803, and a result determination module 804.

[0087] The system comprises the following modules: an acquisition module for acquiring the identification information of the rack under test and the identification information of each node under test within the rack; a binding module for establishing a binding relationship between the rack under test and each node under test based on the identification information of the rack under test and the identification information of each node under test within the rack; a testing module for performing out-of-band testing operations on each node under test based on the out-of-band management channel and the binding relationship between the rack under test and each node under test, in order to obtain the hardware status information fed back by the node under test through the out-of-band management channel; and a result determination module for determining the out-of-band test result of the rack under test based on the hardware status data fed back by each node under test through the out-of-band management channel.

[0088] Specifically, in one embodiment, the binding module is specifically used for: For any node under test in the rack under test, perform a consistency check on the node's identification information. If the identity information of the node under test passes the consistency verification, a binding relationship is established between the server rack under test and the node under test based on the identity information of the server rack under test and the node under test.

[0089] Specifically, in one embodiment, the binding module is specifically used for: For any node under test in the rack under test, the target physical address of the node under test is retrieved from the preset configuration database based on the node's identity serial number; wherein, the identity information of the node under test includes at least the identity serial number. Based on the Dynamic Host Configuration Protocol environment, determine the target logical address to be allocated to the target physical address; Based on the target logical address, obtain the identity sequence verification code of the node to be tested; If the identity sequence check code and identity sequence code of the node under test match, obtain the link layer discovery protocol neighbor information of the node under test; Based on the neighbor information of the link layer discovery protocol, determine the target physical verification address of the node under test; If the target physical verification address of the node under test is consistent with the target physical address, the identity information of the node under test is determined to have passed the consistency verification.

[0090] Specifically, in one embodiment, the binding module is further configured to: If the identity sequence verification code and identity sequence code of the node under test are inconsistent, or if the target physical verification address and target physical address of the node under test are consistent, it is determined that the node under test has a consistency verification anomaly, so as to trigger the process of re-acquiring the identity information of the node under test.

[0091] Specifically, in one embodiment, the test module is specifically used for: The target test time window for the entire cabinet under test; Within the target test time window, based on the out-of-band management channel, and according to the binding relationship between the rack under test and each node under test, the target processor performance test operation is performed on each node under test to obtain the target processor status information fed back by the node under test through the out-of-band management channel. The out-of-band test operations include target processor performance test operations, which include AC power reset operations, and the hardware status information includes target processor status information.

[0092] Specifically, in one embodiment, the test module is specifically used for: Perform a target power supply disturbance operation on the cabinet under test to obtain the defect trigger rate of the cabinet under test in multiple time windows; The time window with the highest defect trigger rate is used as the time window to be corrected. The target test time window for the entire cabinet under test is determined based on the time window to be corrected and the target operation parameters corresponding to the target power supply disturbance operation.

[0093] Specifically, in one embodiment, the test module is specifically used for: The first-level power supply disturbance operation was performed on the entire cabinet under test, and the results of the first-level power supply disturbance operation were obtained; The second-level power supply disturbance operation was performed on the entire cabinet under test, and the results of the second-level power supply disturbance operation were obtained; By combining the results of the first-level power supply disturbance operation and the results of the second-level power supply disturbance operation, the defect trigger rate of the cabinet under test in multiple time windows is determined; The first-level power supply disturbance operation is used to cause a global power supply drop in the cabinet under test, while the second-level power supply disturbance operation is used to cause a local power supply imbalance in the cabinet under test.

[0094] Specifically, in one embodiment, the test module is specifically used for: The target test time window for the entire rack under test is determined based on the following formula:

[0095] in, This indicates the target test time window for the entire rack under test. Indicates the time window to be corrected. This indicates the power supply disturbance recovery speed of the entire cabinet under test. This represents the optimal scaling factor. This indicates the preset correction factor. This indicates the disturbance magnitude; the target operating parameters include the disturbance magnitude.

[0096] Specifically, in one embodiment, the test module is specifically used for: Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, a first fan control command is sent to each test node to control the fan of the test node to rotate at a first speed to generate a first-level mechanical vibration, and the first leakage detection circuit signal fed back by the test node through the out-of-band management channel is obtained. Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, a second fan control command is sent to each test node to control the fan of the test node to rotate at a second speed to generate secondary mechanical vibration, and the second leakage detection loop signal fed back by the test node through the out-of-band management channel is obtained. The second rotational speed is greater than the first rotational speed. The hardware status information includes the first leakage detection circuit signal and the second leakage detection circuit signal. The first leakage detection circuit signal and the second leakage detection circuit signal are used to characterize the health status of the leakage detection line.

[0097] Specifically, in one embodiment, the test module is specifically used for: Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, a global reset operation is performed on each node under test to trigger the rack under test to retrain all target links and obtain the target link topology-level verification information fed back by the node under test through the out-of-band management channel. The hardware status information includes target link topology-level verification information.

[0098] Specifically, in one embodiment, the node under test includes a computing node under test and a switching node under test, and the result determination module is specifically used for: Based on the first link status information fed back by each computing node under test through the out-of-band management channel and the second link status information fed back by the switching node under test through the out-of-band management channel, the first link status information and the second link status information corresponding to the same target link in the rack under test are bidirectionally verified to obtain the bidirectional verification result. Based on the bidirectional verification results of each target link within the rack under test, the test results of the target links of the rack under test are determined. The target link topology-level verification information includes the first link status information and the second link status information, and the out-of-band test results include the target link test results.

[0099] Specifically, in one embodiment, the result determination module is specifically used for: For any target link, if the first link state information of the target link indicates that the link training is successful and the second link state information indicates that the link training is successful, the bidirectional verification result of the target link is determined to be normal. If the first link state information of the target link indicates that the link training has failed, or the second link state information indicates that the link training has failed, the bidirectional verification result of the target link is determined to be abnormal.

[0100] Specifically, in one embodiment, the result determination module is further configured to: If the bidirectional verification result of any target link is abnormal, the port of the computing node under test corresponding to the first link status information and the port of the switching node under test corresponding to the second link status information are marked as abnormal points.

[0101] Specifically, in one embodiment, the result determination module is specifically used for: If the bidirectional verification results of all target links in the rack under test are normal, the test results of the target links in the rack under test are determined to be normal. If the bidirectional verification result of any target link within the rack under test is abnormal, the test result of the target link of the rack under test is determined to be abnormal.

[0102] For a description of the features in the embodiment corresponding to the testing device, please refer to the relevant description of the embodiment corresponding to the testing method, which will not be repeated here.

[0103] Embodiments of this application also provide an electronic device, such as... Figure 9 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to perform the steps in any of the above-described test method embodiments.

[0104] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described test method embodiments when run.

[0105] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0106] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described test method embodiments.

[0107] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described test method embodiments.

[0108] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] The foregoing has provided a detailed description of a testing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A testing method, characterized in that, The method is applied to an out-of-band test host, which is connected in series with the top switch of the rack under test via a network cable. The out-of-band test host establishes an out-of-band management channel with each node under test within the rack under test through the top switch. The method includes: Obtain the identification information of the rack under test and the identification information of each node under test in the rack under test; Based on the identification information of the rack under test and the identification information of each node under test in the rack under test, a binding relationship is established between the rack under test and each node under test. Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, out-of-band testing operations are performed on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel; The out-of-band test results of the rack under test are determined based on the hardware status data fed back by each node under test through the out-of-band management channel.

2. The test method according to claim 1, characterized in that, The step of establishing a binding relationship between the rack under test and each node under test based on the identification information of the rack under test and the identification information of each node under test in the rack under test includes: For any node under test in the rack under test, the consistency of the identification information of the node under test is verified. If the identity information of the node under test passes the consistency verification, a binding relationship is established between the server rack under test and the node under test based on the identity information of the server rack under test and the identity information of the node under test.

3. The test method according to claim 2, characterized in that, The out-of-band test host is equipped with a Dynamic Host Configuration Protocol (DHCP) environment. The process of verifying the consistency of the identity information of any node under test within the rack under test includes: For any node under test in the rack under test, the target physical address of the node under test is retrieved from the preset configuration database based on the identity serial number of the node under test; wherein, the identity identification information of the node under test includes at least the identity serial number; Based on the Dynamic Host Configuration Protocol environment, the target logical address to be allocated to the target physical address is determined. Based on the target logical address, obtain the identity sequence verification code of the node to be tested; If the identity sequence check code of the node under test matches the identity sequence code, obtain the link layer discovery protocol neighbor information of the node under test; The target physical verification address of the node under test is determined based on the link layer discovery protocol neighbor information. If the target physical verification address of the node under test is consistent with the target physical address, the identity information of the node under test is determined to have passed the consistency verification.

4. The test method according to claim 3, characterized in that, The method further includes: If the identity sequence verification code of the node under test is inconsistent with the identity sequence code, or if the target physical verification address of the node under test is consistent with the target physical address, it is determined that the node under test has a consistency verification anomaly, so as to trigger the process of re-acquiring the identity identification information of the node under test.

5. The test method according to claim 1, characterized in that, Based on the out-of-band management channel, and according to the binding relationship between the rack under test and each node under test, out-of-band testing is performed on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel, including: Detect the target test time window of the rack under test; Within the target test time window, based on the out-of-band management channel and according to the binding relationship between the rack under test and each node under test, target processor performance test operations are performed on each node under test to obtain the target processor status information fed back by the node under test through the out-of-band management channel. The out-of-band test operation includes the target processor performance test operation, which includes an AC power reset operation, and the hardware status information includes the target processor status information.

6. The test method according to claim 5, characterized in that, The target test time window for detecting the rack under test includes: A target power supply disturbance operation is performed on the cabinet under test to obtain the defect trigger rate of the cabinet under test in multiple time windows; The time window with the highest defect trigger rate is taken as the time window to be corrected. The target test time window for the entire cabinet under test is determined based on the time window to be corrected and the target operation parameters corresponding to the target power supply disturbance operation.

7. The test method according to claim 6, characterized in that, The step of performing a target power supply disturbance operation on the rack under test to obtain the defect trigger rate of the rack under test in multiple time windows includes: The first-level power supply disturbance operation is performed on the cabinet under test, and the result of the first-level power supply disturbance operation is obtained; A second-level power supply disturbance operation is performed on the cabinet under test, and the result of the second-level power supply disturbance operation is obtained; Based on the combined results of the first-level power supply disturbance operation and the second-level power supply disturbance operation, the defect trigger rate of the cabinet under test is determined in multiple time windows; The first-level power supply disturbance operation is used to cause a global power supply drop in the rack under test, and the second-level power supply disturbance operation is used to cause a local power supply imbalance in the rack under test.

8. The test method according to claim 6, characterized in that, The step of determining the target test time window for the entire rack under test based on the time window to be corrected and the target operation parameters corresponding to the target power supply disturbance operation includes: The target test time window for the entire rack under test is determined based on the following formula: in, This indicates the target test time window for the entire rack under test. This indicates the time window to be corrected. This indicates the power supply disturbance recovery speed of the entire cabinet under test. This represents the optimal scaling factor. This indicates the preset correction factor. The perturbation amplitude is indicated by the target operating parameters.

9. The test method according to claim 1, characterized in that, Based on the out-of-band management channel, and according to the binding relationship between the rack under test and each node under test, out-of-band testing is performed on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel, including: Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, a first fan control command is sent to each test node to control the fan of the test node to rotate at a first speed to generate a first-level mechanical vibration, thereby obtaining the first leakage detection circuit signal fed back by the test node through the out-of-band management channel; Based on the out-of-band management channel, according to the binding relationship between the test cabinet and each test node, a second fan control command is sent to each test node to control the fan of the test node to rotate at a second speed to generate secondary mechanical vibration, thereby obtaining the second leakage detection circuit signal fed back by the test node through the out-of-band management channel; Wherein, the second rotational speed is greater than the first rotational speed, and the hardware status information includes the first leakage detection circuit signal and the second leakage detection circuit signal, which are used to characterize the health status of the leakage detection line.

10. The test method according to claim 1, characterized in that, Based on the out-of-band management channel, and according to the binding relationship between the rack under test and each node under test, out-of-band testing is performed on each node under test to obtain the hardware status information fed back by the node under test through the out-of-band management channel, including: Based on the out-of-band management channel, according to the binding relationship between the rack under test and each node under test, a global reset operation is performed on each node under test to trigger the rack under test to retrain all target links, thereby obtaining the target link topology-level verification information fed back by the node under test through the out-of-band management channel; The hardware status information includes the target link topology-level verification information.

11. The test method according to claim 10, characterized in that, The nodes under test include compute nodes under test and switching nodes under test. Determining the out-of-band test results of the entire rack under test based on the hardware status data fed back by each node under test through the out-of-band management channel includes: Based on the first link status information fed back by each of the computing nodes under test through the out-of-band management channel and the second link status information fed back by the switching nodes under test through the out-of-band management channel, a bidirectional verification is performed on the first link status information and the second link status information corresponding to the same target link in the rack under test to obtain a bidirectional verification result. Based on the bidirectional verification results of each target link within the rack under test, the test results of the target links of the rack under test are determined. The target link topology-level verification information includes the first link status information and the second link status information, and the out-of-band test results include the target link test results.

12. The test method according to claim 11, characterized in that, The first link status information and the second link status information corresponding to the same target link within the rack under test are bidirectionally verified based on the first link status information fed back by each of the computing nodes under test through the out-of-band management channel and the second link status information fed back by the switching nodes under test through the out-of-band management channel, to obtain the bidirectional verification result, including: For any of the target links, if the first link state information of the target link indicates that the link training is successful and the second link state information indicates that the link training is successful, the bidirectional verification result of the target link is determined to be normal. If the first link state information of the target link indicates that the link training has failed, or the second link state information indicates that the link training has failed, the bidirectional verification result of the target link is determined to be abnormal.

13. The test method according to claim 12, characterized in that, The method further includes: If the bidirectional verification result of any of the target links is abnormal, the test computing node corresponding to the first link status information and the test switching node port corresponding to the second link status information are marked as abnormal points.

14. The test method according to claim 12, characterized in that, The step of determining the target link test results of the rack under test based on the bidirectional verification results of each target link within the rack under test includes: If the bidirectional verification results of all target links in the rack under test are normal, the test results of the target links in the rack under test are determined to be normal. If the bidirectional verification result of any target link within the rack under test is abnormal, the test result of the target link of the rack under test is determined to be abnormal.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the test method as described in any one of claims 1 to 14.

Images