A method for performance and stress testing of computing nodes and electronic devices.

Abstract

This invention discloses a method and electronic device for performance and stress testing of computing nodes, relating to the field of computer hardware testing technology. In this method, a target test configuration file is determined based on the topology of the computing node, and target test items are determined based on the selection operation of test items. Then, a target test tool and a target test mode are determined based on the target test configuration file and sent to the computer device under test. Finally, the test tool is used for automated performance and stress testing. The test mode can be flexibly changed based on the test items. Furthermore, the test configuration file integrates the test items and test tools, avoiding fragmentation of test tools, and considers the topology of the computing node, enabling the determined test strategy to adapt to different hardware environments. Moreover, when the target test items are different, the determined test tools are different, and correspondingly, the test tools received by the computer device under test are also different, thus achieving automatic switching of test tools.

Description

Technical Field

[0001] This invention relates to the field of computer hardware testing technology, and more particularly to a method for testing the performance and stress of computing nodes and electronic devices. Background Technology

[0002] Graphics Processing Units (GPUs) possess powerful parallel computing capabilities. To improve GPU reliability, GPUs need to be tested.

[0003] In practice, although there are various GPU testing tools available, each tool has relatively limited functionality, meaning different tools are required for different test items. When testing different items, users often have to manually switch between different tools, which is cumbersome and inefficient.

[0004] Therefore, improving testing efficiency is a technical problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] This invention provides a method and electronic device for testing the performance and stress of computing nodes, which at least solves the problem in related technologies that users often have to manually switch between different testing tools when testing different test items, which is cumbersome and inefficient.

[0006] This invention provides a method for performance and stress testing of computing nodes, applied to execution machines, the method comprising:

[0007] Create test configuration files based on the test items, the corresponding test tools, and the test modes; among them, the test modes include at least the single-component polling test mode and the multi-component parallel test mode;

[0008] Obtain the mapping relationship between the topology of the compute nodes and the test configuration file;

[0009] When it detects that it is connected to the computer device under test, it determines the target test configuration file corresponding to the topology of the computing nodes in the computer device under test according to the mapping relationship, and determines the target test items corresponding to the computing nodes in the computer device under test according to the test item selection operation on the user interface.

[0010] Based on the target test configuration file, determine the target test tool and target test mode corresponding to the target test item, and send the target test tool and target test mode to the computer device under test so that the target test tool can perform the test on the computer device under test;

[0011] Obtain the test results of the target test tool on the target test item under the target test mode.

[0012] The beneficial effects of this invention are as follows: First, in this method, the executor creates a test configuration file (recording the relationship between test items, test tools, and test modes); and after the computer device under test is connected, it determines the target test configuration file based on the topology of the computing nodes, and determines the target test items based on the selection operation of test items on the user interface; then, it determines the target test tool and target test mode based on the target test configuration file, and sends the target test tool and test mode to the computer device under test, ultimately using the test tool to realize automated testing of the performance and stress of the computing nodes of the computer device under test, and the test mode can be flexibly changed based on the test items; Second, the test configuration file created by the executor integrates test items and test tools, avoiding fragmentation of test tools, and the mapping relationship established based on the topology of the computing nodes and the test configuration file takes into account the topology of the computing nodes, so that the test strategy selected according to the mapping file can adapt to different hardware environments; Third, after the executor is connected to the computer device under test, it finds the test tool and test mode corresponding to the target test item based on the mapping relationship, and sends the target test tool and target test mode to the computer device under test. When the target test items are different, the test tools determined according to the mapping relationship are different. Correspondingly, the test tools received by the computer device under test are also different. That is, the test tools are automatically switched when testing different test items, without the need for manual switching by the user. In addition, the target test items are determined based on the selection operation of test items on the user interface. That is, the method can meet the user's testing needs.

[0013] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the performance and stress testing method of any of the above-described computing nodes when executing the computer program. Attached Figure Description

[0014] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram of a test scenario provided in an embodiment of the present invention;

[0016] Figure 2 A flowchart illustrating a method for testing the performance and stress of a computing node, as provided in an embodiment of the present invention;

[0017] Figure 3 A flowchart illustrating a method for predicting computing node state data provided in an embodiment of the present invention;

[0018] Figure 4 A flowchart illustrating a monitoring and control testing method provided in an embodiment of the present invention;

[0019] Figure 5 This is an overall flowchart of a method for obtaining the performance and stress test results of a computing node, provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0021] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. First, the test scenario of the present invention will be described. Figure 1 This is a schematic diagram of a test scenario provided in an embodiment of the present invention, such as... Figure 1 As shown, the test scenario includes an execution machine 1 and a computer device under test 2 (hereinafter referred to as the test machine). The execution machine 1 is connected to the computer device under test 2. The performance and stress testing method for computing nodes provided by this invention is applied to the execution machine. Figure 2 A flowchart of a performance and stress testing method for a computing node provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0023] S10: Create test configuration files based on the test items, the corresponding test tools, and the test modes; among them, the test modes include at least the single-component polling test mode and the multi-component parallel test mode;

[0024] S11: Obtain the mapping relationship between the topology of the compute nodes and the test configuration file;

[0025] S12: When it is detected that it is connected to the computer device under test, the target test configuration file corresponding to the topology of the computing node in the computer device under test is determined according to the mapping relationship, and the target test item corresponding to the computing node in the computer device under test is determined according to the test item selection operation on the user interface.

[0026] S13: Determine the target test tool and target test mode corresponding to the target test item according to the target test configuration file, and send the target test tool and target test mode to the computer device under test so that the target test tool can perform the test on the computer device under test;

[0027] S14: Obtain the test results of the target test tool on the target test item under the target test mode.

[0028] If the computing node is a GPU, the performance test of the computing node in this invention refers to the test of a single computing node; the stress test of the computing node refers to the parallel test of multiple computing nodes. The test items are not limited and are determined according to the actual situation. For example, in performance testing, test items include computing power and bandwidth; in stress testing, test items include stability and power consumption. The testing tools used to test the test items and the test modes used to test the test items are obtained. The test modes include at least a single-component polling test mode and a multi-component parallel test mode. The single-component polling test mode can be used to test the performance of the computing node; the multi-component parallel test mode can be used to test the stress of the computing node.

[0029] After obtaining the test items, corresponding test tools, and test modes, a configuration file is created based on these information. During implementation, the number of tests can also be set in the test configuration file. Furthermore, to adapt the testing strategy to different hardware environments, the topology of the compute nodes (i.e., hardware information), including interconnection methods, the number of compute nodes, and physical layout, is also obtained during implementation. A mapping relationship is then established between the compute node topology and the test configuration file.

[0030] After storing the mapping relationship in the execution machine, if the execution machine detects that it is connected to a computer device (to be tested; therefore, in this embodiment of the invention, the computer device connected to the execution machine is referred to as the computer device under test), it first obtains the topology of the computing nodes in the computer device under test, and then finds the test configuration file corresponding to the topology from the mapping relationship. This found test configuration file is called the target test configuration file. The test items, the corresponding test tools, and the test modes can be found through the target test configuration file.

[0031] In practice, users have different testing needs. Therefore, to meet these needs, this invention incorporates a user interface into the execution machine. A graphical interface or command-line interface is provided, allowing users to select multiple test items, such as performance tests (computing power, bandwidth) and stress tests (stability, power consumption). Users can set the execution order of the test items or use the default recommended execution order. The target test item corresponding to the computing node in the computer device under test is determined based on the test item selection operation on the user interface.

[0032] After identifying the target test item and determining the corresponding target test tool and target test mode based on the target test configuration file, the execution machine sends the target test tool and target test mode to the computer device under test. The target test tool performs the test on the computer device under test. Simultaneously, the execution machine is also configured to monitor the test execution status of the test tool on the computer device under test. When it detects that the test tool has completed the test on the computer device under test, it retrieves the test results of the target test tool on the target test item under the target test mode.

[0033] In the performance and stress testing method for computing nodes provided in this embodiment, firstly, the execution machine creates a test configuration file (recording the relationship between test items, test tools, and test modes); after the computer device under test is connected, it determines the target test configuration file based on the topology of the computing node, and determines the target test items based on the selection operation of test items on the user interface; then, it determines the target test tool and target test mode based on the target test configuration file, and sends the target test tool and test mode to the computer device under test, and finally uses the test tool to realize the automated testing of the performance and stress of the computing node of the computer device under test, and the test mode can be flexibly changed based on the test items; secondly, the test configuration file created by the execution machine realizes the integration of test items and test tools, avoids the fragmentation of test tools, and the mapping relationship established based on the topology of the computing node and the test configuration file takes into account the topology of the computing node, so that the test strategy selected according to the mapping file can adapt to different hardware environments; thirdly, after the execution machine is connected to the computer device under test, it finds the test tool and test mode corresponding to the target test item based on the mapping relationship, and sends the target test tool and target test mode to the computer device under test. When the target test items are different, the test tools determined according to the mapping relationship are different. Correspondingly, the test tools received by the computer device under test are also different. That is, the test tools are automatically switched when testing different test items, without the need for manual switching by the user. In addition, the target test items are determined based on the selection operation of test items on the user interface. That is, the method can meet the user's testing needs.

[0034] The computer device under test includes multiple computing nodes, and determining a suitable testing order for the computing nodes improves the reliability of the test results. Therefore, in some embodiments, before sending the target testing tool and target testing mode to the computer device under test, the following steps are also included:

[0035] The external communication connection method of the computing nodes is determined based on the topology of the computing nodes in the computer device under test; wherein, the external communication connection method includes at least dedicated interconnection method and general bus connection method;

[0036] The test order of computing nodes in the computer device under test is determined based on the external communication connection method of the computing nodes.

[0037] Sending the target testing tool and target testing mode to the computer device under test so that the target testing tool can perform testing on the computer device under test includes:

[0038] The target testing tool, target testing mode, and test sequence are sent to the computer device under test, so that the target testing tool can perform the target test items corresponding to the computing nodes in the computer device under test in the test sequence and under the target testing mode.

[0039] In this method, the testing order of computing nodes in the computer device under test is determined based on the external communication connection method of the computing nodes. The external communication connection method includes at least dedicated interconnection methods and general-purpose bus connection methods. Dedicated interconnection methods can achieve low-latency, high-bandwidth communication between computing nodes, which is suitable for parallel computing tasks with multiple computing nodes and can significantly improve performance; general-purpose bus (such as PCIe) connection methods have broad compatibility and are suitable for single computing nodes or scenarios where the communication requirements between multiple computing nodes are not high.

[0040] When testing compute nodes, the nodes connected via dedicated interconnects are tested first, followed by those connected via a general-purpose bus. Firstly, dedicated interconnects are interfaces specifically designed for high-speed communication between compute nodes, offering significantly higher bandwidth than general-purpose bus connections. This enables more efficient data transmission and collaboration between compute nodes. Therefore, testing compute nodes connected via dedicated interconnects first provides a more intuitive assessment of their performance in high-performance computing scenarios, especially when multiple compute nodes need to process tasks in parallel, quickly identifying potential performance bottlenecks. Secondly, the low latency of dedicated interconnects reduces synchronization overhead when multiple compute nodes collaborate. Testing this first better verifies the collaborative efficiency of compute nodes in complex computing tasks, providing a more accurate reference for subsequent optimization. Finally, logically, testing the more powerful dedicated interconnects first provides a performance reference for subsequent testing of general-purpose bus-connected compute nodes, facilitating comparative analysis and a more comprehensive understanding of the performance differences between compute nodes using different connection methods. This provides a strong basis for users to select appropriate compute node configurations.

[0041] To understand the testing situation and improve the efficiency of transmitting test results, the performance and stress testing methods for computing nodes also include:

[0042] During the detection process of the target testing tool performing tests on the computer device under test, the topology information of the target computing node currently being tested is obtained; wherein, the topology information includes at least the correspondence between the target computing node and the non-unified memory access architecture node, and the communication link attributes between the target computing node and the non-unified memory access architecture node;

[0043] Candidate communication paths are obtained based on the topology information of the target computing node;

[0044] Obtain the transmission characteristics of candidate communication paths; wherein, the transmission characteristics include at least latency characteristics and transmission characteristics across nodes of non-uniform memory access architecture;

[0045] Based on the transmission characteristics of the candidate communication paths, the target candidate communication path with the minimum latency and which does not have the transmission characteristics of cross-node non-uniform memory access architecture is selected from the candidate communication paths.

[0046] The status data of the computing node is obtained through the target candidate communication path; the status data includes at least the core temperature, fan speed, power consumption, memory usage or error correction code count.

[0047] If the detected status data is greater than the preset value corresponding to the status data, output a prompt message to characterize the test failure of the computing node.

[0048] The preset values ​​are not limited and are determined based on the actual situation. In this method, by monitoring the status of the computing nodes during the test and obtaining the status data of the computing nodes, the user can understand the status of the computing nodes during the test; in addition, the status data of the computing nodes is transmitted through a communication path with minimal latency and without crossing non-uniform memory access architecture nodes, which improves data transmission efficiency and reduces data transmission costs.

[0049] To determine in advance whether a computing node has malfunctioned during testing, in practice, after obtaining the computing node's status data through the target candidate communication path, and before outputting a prompt message indicating a computing node test failure if the status data exceeds a preset value, the following steps are also included:

[0050] The acquired state data of the computing nodes is used as historical state data;

[0051] Obtain the first historical state data within the first preset time period and obtain the prediction time step;

[0052] Acquire the second historical state data within a preset time step, starting from the end of the first preset time.

[0053] The first historical state data is used as the input sequence and the second historical state data is used as the output to train a deep learning model.

[0054] Input the state data of the computing nodes within a preset time step before the time to be predicted into the trained deep learning model;

[0055] The trained deep learning model outputs the state data of the computation nodes within a preset time step, starting from the time to be predicted.

[0056] There are no restrictions on the first preset time and the predicted time step; they are determined based on the actual situation. Figure 3 A flowchart of a method for predicting computation node state data provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method includes:

[0057] S15: Acquire sensor data;

[0058] S16: Feature extraction based on sensor data;

[0059] S17: Prediction using deep learning models;

[0060] S18: Obtain the predicted state data of the computing nodes.

[0061] Table 1 shows the input feature table. It records some features and corresponding data examples for some features. Among them, Timestamp represents the timestamp, gpu_temp represents the GPU temperature, Power represents the power, ecc_errors represents checksum errors, Clock represents the clock frequency, and Util represents the utilization rate.

[0062] Table 1

[0063]

[0064] Predicting the state in the next 5 minutes using a deep learning model:

[0065] class FailurePredictor:

[0066] def predict(self,sensor_data):

[0067] with self.model.as_default():

[0068] return self.model.predict(

[0069] np.array([sensor_data]).reshape(1, 60, 5) # 60 time steps * 5 features.

[0070] This method uses historical state data of computing nodes to predict future state data of computing nodes, thereby determining in advance whether computing nodes will fail during the test based on the predicted state data.

[0071] During testing, a compute node failure may be detected. To allow for continued testing of other compute nodes in the computer device under test, the implementation includes, after outputting a prompt message indicating a compute node test failure, the following:

[0072] The faulty computing node is identified based on the prompt information used to characterize the test failure of the computing node;

[0073] Obtain the current test mode corresponding to the faulty computing node;

[0074] Based on the current testing mode, different strategies are selected to migrate the corresponding test information when testing test items; among which, the test information includes the corresponding runtime environment or test task during testing.

[0075] In this method, if a computing node fails during testing, the test task corresponding to the test item is migrated, ensuring automatic switching of tests in the event of a failure.

[0076] In some embodiments, the migration of test information corresponding to different strategies selected for testing test items based on the current testing mode includes:

[0077] If the current test mode is detected to be a single compute node polling test mode, obtain the locked memory pages in the faulty compute node; the locked memory pages record the current running environment corresponding to the test item during the test.

[0078] Obtain the next computing node to be tested after the faulty computing node according to the testing order of the computing nodes in the computer device under test;

[0079] The information in the locked memory pages of the faulty compute node is migrated to the next compute node to be tested, so that the next compute node to be tested can determine the serial number of the compute node from which the information came, and the target test tool is used to test the target test item corresponding to the next compute node to be tested.

[0080] In the automatic failover algorithm, an automatic detection and failover mechanism is designed for single-compute node testing. Once a failure is detected in the current single compute node, a memory page is locked via CUDA UVM, the context is transferred across devices using GPUDirect RDMA, and the compute flow is re-bound to another compute node to continue testing. This ensures the failover process is transparent to the user and does not affect the continuity and integrity of the test. Specifically, upon detecting a failure in the current compute node, the failed compute node will have a locked memory block (the locked memory block records the current runtime environment). After the failure, it migrates its locked information to another compute node (the other compute node can know the compute node from which the migrated information originated based on the sequence number of the failed compute node). This method allows the compute node to which the information is migrated to to directly use the information migrated from the failed compute node without needing to obtain runtime environment information itself, thus improving testing efficiency. Furthermore, since only a locked memory block is used (i.e., no new data is generated), data integrity and consistency are guaranteed.

[0081] If the current test mode is detected to be a multi-compute node parallel test mode, obtain the total number of compute nodes in the computer device under test and the number of compute nodes called; wherein, the compute nodes called are all compute nodes under test.

[0082] If the number of compute nodes called is less than the total number of compute nodes, the test task corresponding to the test item of the faulty compute node is paused.

[0083] Obtain the computing node to be migrated from the remaining computing nodes; where the remaining computing nodes are the nodes in the computer device under test other than the computing node to be called.

[0084] The test tasks corresponding to the test items of the faulty computing node are migrated to the computing node to be migrated, so that the target test tools can be used to test the target test items corresponding to the computing node to be migrated.

[0085] In this method, under the multi-computing node parallel testing mode, automatic switching of test tasks is realized, ensuring that the scheduled computing nodes meet the requirements, and improving the reliability of the scheduled computing nodes through testing.

[0086] The test task migration described above ensured that the test could continue. This embodiment ensures that the test can continue by reducing the frequency. Specifically, after outputting the prompt information characterizing the test failure of the computing node, it also includes:

[0087] Obtain the current operating frequency of the faulty computing node;

[0088] Adjust the current operating frequency to the preset operating frequency; wherein the preset operating frequency is lower than the current operating frequency;

[0089] If the operating frequency of the faulty computing node is detected to be within the preset operating frequency, the target test item corresponding to the computing node in the computer device under test is tested.

[0090] In this method, after identifying the faulty computing node, the frequency of the faulty computing node is reduced, thereby ensuring that the testing of the computing node can continue as much as possible.

[0091] Figure 4 A flowchart of a monitoring and control testing method provided in an embodiment of the present invention is shown below. Figure 4 As shown, the method includes:

[0092] S19: Acquire sensor data;

[0093] S20: Extract features from sensor data;

[0094] S21: Perform anomaly detection and identify the anomaly;

[0095] S22: Task migration;

[0096] S23: Dynamic frequency reduction;

[0097] S24: Continue testing.

[0098] This method enables automatic switching between fault detection, handling, and testing.

[0099] Because the differences between computer devices under test (DUTs) and those operating systems are significant, they are typically adapted separately for each DUT. Furthermore, different operating systems support different test result formats. Therefore, to improve test compatibility and facilitate comparison of test results, in some embodiments, the DUT is a computer device with a different operating system; connecting to the DUT via a unified interface includes connecting to different DUTs using the same interface.

[0100] In the target testing mode, after obtaining the test results of the target test item by the target testing tool, it also includes:

[0101] The format of the test results should be determined based on the operating system of the computer device under test;

[0102] The test results of the computer devices under test with different operating systems are converted into test results in the same format.

[0103] This method uses a unified testing interface for different operating systems and standardizes the test format to facilitate comparison of results and ensure cross-platform compatibility.

[0104] To help users intuitively understand the test results, after obtaining the test results of the target test item in the target test mode, the target test tool also includes:

[0105] The test results are analyzed to obtain a report on the performance metrics and stability assessment results used to characterize the computing nodes;

[0106] The operating parameters to be adjusted for the computing nodes are determined based on the reports of performance metrics and stability assessment results used to characterize the computing nodes.

[0107] In this method, after obtaining the test results, a report is generated, allowing the user to identify the content to be optimized.

[0108] The aforementioned performance and stress testing methods for computing nodes can be implemented through a computing node performance and stress testing system. This system includes a main control module, a test execution module, a cross-platform interface module, a monitoring module, an intelligent analysis module, a fault switching module, and a result analysis module. The function of each module is explained below.

[0109] 1. Main Control Module: Responsible for the control and management of the entire testing process, including test item selection (such as stream, p2p, PeakTops, cublasMatmulBench, fieldiag, DCGM, gpu-burn, etc.), execution order settings, multi-node parallel testing coordination, and fault handling decisions. Dynamic Loading Mechanism: Designed with a plug-in architecture (i.e., the test configuration file described above), allowing new testing tools or test items to be integrated into the system through simple configuration or development.

[0110] 2. Test Execution Module: Includes a multi-mode test engine, supporting single-node polling tests and multi-node parallel tests. It can call corresponding test components according to different test items, such as performance test components and stress test components.

[0111] Automatic hardware topology detection acquires real-time hardware connection methods and physical layout of computing nodes, generates a topology map, automatically selects the corresponding testing tools based on the topology, sets test parameters (such as test mode or number of tests), and intelligently adjusts the testing strategy to adapt to different hardware environments.

[0112] 3. Cross-platform interface module: Enables compatibility and adaptation across multiple platforms, unifies test interfaces and data formats, and ensures consistency and comparability of tests on different platforms.

[0113] 4. Monitoring module: Collects various status data of computing nodes in real time, such as core utilization, temperature, power consumption, ECC errors, etc., and performs real-time analysis and early warning.

[0114] 5. Intelligent Analysis Module: Based on monitoring data, it automatically identifies critical values ​​of computing node fault states, intelligently analyzes potential problems based on test results, and provides solutions.

[0115] 6. Fault Switching Module: Implements an automatic switching algorithm when a single computing node fails during testing, ensuring the continuity and reliability of testing.

[0116] 7. Results Analysis Module: Automatically collects and analyzes test data, and generates detailed test reports, including but not limited to performance indicators, stability assessments, temperature and frequency curves, and error logs.

[0117] Figure 5 This invention provides an overall flowchart of a method for obtaining performance and stress testing results for computing nodes, as shown in the embodiments of the present invention. Figure 5 As shown, the method includes:

[0118] S25: Initialization and Environment Configuration;

[0119] S26: Test item selection and execution order settings;

[0120] S27: Multi-mode test execution;

[0121] S28: Real-time status monitoring;

[0122] S29: Determine if a fault has occurred; if yes, proceed to step S30; if no, proceed to step S31.

[0123] S30: Fault handling and automatic switching;

[0124] S31: Test complete;

[0125] S32: Results analysis and suggestion generation.

[0126] Specifically, the methods for performing performance and stress tests on compute nodes include:

[0127] Initialization and environment configuration:

[0128] Establish a communication connection between the actuator and the computer device under test to ensure normal communication.

[0129] Identify the type of system platform and load the corresponding cross-platform adapter driver.

[0130] Hardware topology detection: Real-time acquisition of hardware information for compute nodes, including interconnection methods, number of compute nodes, and physical layout. Based on the acquired topology information, a hardware configuration diagram is automatically generated, and appropriate testing tools and parameters are intelligently selected accordingly.

[0131] When it is detected that some computing nodes use a dedicated interconnect method while others use a general-purpose bus connection method, the test method is as follows:

[0132] def optimize_test_plan(topology):

[0133] if topology.has_asymmetric_nvlink():

[0134] # Prioritize performing high-bandwidth testing within dedicated interconnect groups;

[0135] for island in topology.nvlink_islands():

[0136] assign_bandwidth_test(island);

[0137] #Universal bus connection to compute nodes performs latency-sensitive tests;

[0138] assign_latency_test(topology.pcie_gpus()).

[0139] Test item selection and execution order settings:

[0140] It provides a graphical interface or command-line interface for users to select multiple test items, such as performance tests (computing power, bandwidth), stress tests (stability, power consumption), etc.

[0141] Allows users to set the execution order of test items, or use the default recommended execution order.

[0142] Multi-mode test execution:

[0143] Automated scheduling: The task scheduler can automatically generate test plans based on user-selected test items and compute node configurations. It supports adjusting the order of test items and setting the execution mode for each test item.

[0144] 1. Single compute node polling test: Each compute node is tested sequentially, which is suitable for detailed single compute node performance analysis.

[0145] 2. Parallel testing of multiple computing nodes: Simultaneously test multiple computing nodes, support resource contention management, rationally allocate system resources, and avoid resource conflicts.

[0146] Real-time status monitoring:

[0147] The monitoring module collects real-time status data of the computing nodes, including core temperature, fan speed, power consumption, memory usage, ECC error count, etc.

[0148] Set normal ranges and thresholds for each indicator, and trigger an early warning mechanism when data exceeds the threshold.

[0149] Topology-aware communication optimization: Automatically selects the optimal communication path in p2p tests to avoid transmission across NUMA nodes.

[0150] An intelligent resource scheduler is introduced to achieve dynamic load balancing based on deep reinforcement learning: a deep reinforcement learning model is built to adjust processor utilization in real time, adjust test load, and optimize resource allocation.

[0151] Fault handling and automatic switching:

[0152] When a computing node malfunctions during testing, such as overheating or exceeding the ECC error rate limit, the fault switching module automatically identifies the fault and switches the test task to another normal computing node to continue execution.

[0153] Record information about the faulty computing nodes and the fault type to facilitate subsequent analysis and maintenance.

[0154] Fault Detection and Diagnosis: The anomaly prediction subsystem can monitor system logs and test data in real time to identify potential fault points, such as overheating and performance degradation. It uses predictive models to train machine learning algorithms for fault diagnosis, providing possible causes and suggested solutions.

[0155] Results analysis and suggestion generation:

[0156] The intelligent analysis module uses a data collector to automatically gather test data from various testing tools and store it in a unified database. Data analysis algorithms are then used to deeply analyze the test data, generating reports on performance metrics, stability assessments, and more. Potential problems, such as performance bottlenecks and poor heat dissipation, are automatically identified, and corresponding solutions are provided, such as optimizing the cooling system and adjusting the operating frequency.

[0157] The report features visually appealing charts and graphs to intuitively display test results, such as temperature and frequency change curves, and performance comparison graphs. It also displays fault records and suggestions, and supports report export and sharing functions for easy further analysis and discussion.

[0158] After the computing node test of the computer device under test is completed, the following is also included:

[0159] Terminate all processes related to the testing tools;

[0160] Delete the test tool files from the storage device, as well as all temporary and log files generated during the test;

[0161] Clean up the resources used on the computing nodes during the testing process;

[0162] Obtain information on system resource usage and determine whether to completely recycle the testing tools based on resource usage.

[0163] This method deletes all temporary and log files generated during the test to avoid occupying unnecessary storage space; and by releasing computing node resources, it ensures that the computing nodes are restored to their initial state as much as possible.

[0164] The above process will be explained below using single-node performance testing and multi-node stress parallel testing as examples.

[0165] I. The single compute node performance testing method includes the following steps:

[0166] 1. Initialization and Environment Configuration: Under the operating system, start the test device, establish a communication connection between the executor and the computer device under test, load the operating platform adapter driver, and scan for a pre-defined computing node installed in the system.

[0167] 2. Test Item Selection and Execution Order Settings: Users can select computing power test (CUBLAS matrix multiplication) and bandwidth test (h2d data transmission) through the graphical interface, and set the computing power test to be performed first, followed by the bandwidth test.

[0168] 3. Multi-mode test execution: A single-node polling mode is adopted to sequentially perform computing power and bandwidth tests on the computing node. The test execution module calls the corresponding test components, and the main control module controls the test order.

[0169] 4. Real-time Status Monitoring: The monitoring module collects data such as temperature, power consumption, and memory usage of the computing nodes in real time. During the computing power test, an alarm is triggered when the temperature reaches 85℃ (the critical value is 90℃), but the test continues as long as the critical value is not exceeded.

[0170] 5. Fault handling and automatic switching: No fault occurred during the test, so no switching was required.

[0171] 6. Results Analysis and Recommendation Generation: The intelligent analysis module analyzes the test results. The computing power test result reaches 95% of the theoretical value, and the bandwidth test result reaches 92% of the nominal value. The analysis concludes that the performance is good, and it is recommended to regularly clean the computing node cooling fans to maintain a good heat dissipation environment. A detailed test report is generated.

[0172] II. Parallel stress testing of multiple computing nodes includes the following steps:

[0173] 1. Initialization and Environment Configuration: Under the operating system, start the test device, establish a communication connection, load the operating platform adapter driver, and scan for the four preset computing nodes installed in the system.

[0174] 2. Test item selection and execution order settings: Users select stability stress test and power consumption test, and set to perform parallel tests on 4 computing nodes simultaneously.

[0175] 3. Multi-mode test execution: The test execution module performs stress tests on four computing nodes simultaneously, while the main control module manages resource contention and allocates CPU and memory resources reasonably to avoid resource conflicts.

[0176] 4. Real-time Status Monitoring: The monitoring module collects data such as temperature, power consumption, and ECC error count for each compute node in real time. During the test, the temperature of the second compute node rapidly rose to 95℃ (the critical value is 90℃), and the ECC error count exceeded the threshold, triggering a critical warning.

[0177] 5. Fault Handling and Automatic Switching: When the fault switching module detects a fault in the second compute node, it immediately suspends the test tasks on that compute node, records the fault information, and migrates the unfinished test tasks to the fourth normal compute node for continued execution.

[0178] 6. Results Analysis and Recommendation Generation: After the test, the intelligent analysis module analyzed the results and found that the second compute node may have a heat dissipation system failure. It recommended checking the heatsink and fan, and repairing or replacing them if necessary. The other three compute nodes performed stably under stress testing, with power consumption within the normal range. A test report containing fault handling records and recommendations was generated.

[0179] The method provided by this invention supports automated integrated testing of computing node performance and stress across platforms and multiple modes, solves the problem of fragmented testing processes, improves the testing efficiency of multi-computing node systems, ensures testing security, and reduces hardware failure rates.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0181] Embodiments of the present invention also provide a performance and stress testing apparatus for a computing node, comprising:

[0182] The module is used to create test configuration files based on test items, corresponding test tools, and test modes; among them, the test modes include at least single-component node polling test mode and multi-component node parallel test mode;

[0183] The first acquisition module is used to acquire the mapping relationship between the topology of the computing nodes and the test configuration file;

[0184] The first determining module is used to determine the target test configuration file corresponding to the topology of the computing nodes in the computer device under test according to the mapping relationship when it detects that it is connected to the computer device under test, and to determine the target test item corresponding to the computing node in the computer device under test according to the selection operation of the test item on the user interface.

[0185] The sending module is used to determine the target test tool and target test mode corresponding to the target test item according to the target test configuration file, and send the target test tool and target test mode to the computer device under test so that the target test tool can perform the test on the computer device under test;

[0186] The second acquisition module is used to acquire the test results of the target test tool on the target test item under the target test mode.

[0187] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0188] The second determining module is used to determine the external communication connection method of the computing node based on the topology of the computing node in the computer device under test; wherein, the external communication connection method includes at least a dedicated interconnection method and a general bus connection method;

[0189] The third determination module is used to determine the test order of computing nodes in the computer device under test based on the external communication connection method of the computing nodes;

[0190] The sending module includes:

[0191] The sending submodule is used to send the target testing tool, target testing mode, and test sequence to the computer device under test, so that the target testing tool can perform the target test items corresponding to the computing nodes in the computer device under test in the test sequence and under the target testing mode.

[0192] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0193] The third acquisition module is used to acquire the topology information of the target computing node being tested when the target testing tool is detected to be testing the computer device under test; wherein, the topology information includes at least the correspondence between the target computing node and the non-unified memory access architecture node, and the communication link attributes between the target computing node and the non-unified memory access architecture node.

[0194] The fourth acquisition module is used to acquire candidate communication paths based on the topology information of the target computing node;

[0195] The fifth acquisition module is used to acquire the transmission characteristics of candidate communication paths; wherein, the transmission characteristics include at least latency characteristics and transmission characteristics across nodes of a non-uniform memory access architecture.

[0196] The selection module is used to select the target candidate communication path with the minimum latency and non-cross-non-uniform memory access architecture node transmission characteristics from the candidate communication paths based on the transmission characteristics of the candidate communication paths.

[0197] The sixth acquisition module is used to acquire the status data of the computing node through the target candidate communication path; wherein, the status data includes at least the core temperature, fan speed, power consumption, memory usage or error correction code count;

[0198] The first output module is used to output a prompt message to characterize a test failure of the computing node when the detected status data is greater than the preset value corresponding to the status data.

[0199] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0200] As a module, it is used to use the acquired state data of computing nodes as historical state data;

[0201] The seventh acquisition module is used to acquire the first historical state data within a first preset time period and to acquire the prediction time step.

[0202] The eighth acquisition module is used to acquire the second historical state data within a preset time step, starting from the end of the first preset time.

[0203] The training module is used to train a deep learning model by taking the first historical state data as the input sequence and the second historical state data as the output.

[0204] The input module is used to input the state data of the computing nodes within a preset time step before the time to be predicted into the trained deep learning model;

[0205] The second output module is used to output the state data of the computation nodes within a preset time step starting from the time to be predicted, based on the trained deep learning model.

[0206] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0207] The fourth determination module is used to determine the faulty computing node based on the prompt information used to characterize the test failure of the computing node;

[0208] The ninth module is used to obtain the current test mode corresponding to the faulty computing node;

[0209] The migration module is used to select different strategies to migrate the test information corresponding to the test items during testing, based on the current test mode; the test information includes the runtime environment or test task corresponding to the test.

[0210] In some embodiments, the migration module includes:

[0211] The tenth acquisition module is used to acquire the locked memory pages in the faulty computing node when the current test mode is detected to be a single computing node polling test mode; wherein, the locked memory pages record the current running environment corresponding to the test item during the test;

[0212] The eleventh acquisition module is used to acquire the next computing node to be tested after the faulty computing node according to the test order of the computing nodes in the computer device under test;

[0213] The first migration submodule is used to migrate the information in the locked memory pages of the faulty compute node to the next compute node to be tested; so that the next compute node to be tested can determine the serial number of the compute node from which the information comes, and use the target testing tool to test the target test item corresponding to the next compute node to be tested.

[0214] The twelfth acquisition module is used to acquire the total number of computing nodes in the computer device under test and the number of computing nodes called when the current test mode is detected to be a multi-computing node parallel test mode; wherein, the computing nodes called are all computing nodes under test.

[0215] The pause module is used to pause the test task corresponding to the test item of the faulty compute node if it is detected that the number of compute nodes called is less than the total number of compute nodes.

[0216] The thirteenth acquisition module is used to acquire the computing node to be migrated from the remaining computing nodes; wherein, the remaining computing nodes are the nodes in the computer device under test other than the computing node to be called.

[0217] The second migration submodule is used to migrate the test tasks corresponding to the test items of the faulty computing node to the computing node to be migrated, so that the target test tools can be used to test the target test items corresponding to the computing node to be migrated.

[0218] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0219] The fourteenth acquisition module is used to acquire the current operating frequency of the faulty computing node;

[0220] The adjustment module is used to adjust the current operating frequency to a preset operating frequency; wherein the preset operating frequency is lower than the current operating frequency.

[0221] The testing module is used to test the target test items corresponding to the computing nodes in the computer device under test when the operating frequency of the detected faulty computing node is at a preset operating frequency.

[0222] In some embodiments, the computer device under test is a computer device with a different operating system; connecting itself to the computer device under test includes: connecting different computer devices under test through its own unified interface;

[0223] The performance and stress testing apparatus for compute nodes also includes:

[0224] The fifth determination module is used to determine the format of the test results based on the operating system of the computer device under test;

[0225] The conversion module is used to convert test results from computer devices under test with different operating systems into test results in the same format.

[0226] In some embodiments, the performance and stress testing apparatus for computing nodes further includes:

[0227] The analysis module is used to analyze the test results to obtain reports on performance metrics and stability assessments for the computing nodes.

[0228] The sixth determination module is used to determine the operating parameters of the computing node to be adjusted based on the report of performance indicators and stability evaluation results used to characterize the computing node.

[0229] For a description of the features in the embodiment corresponding to the performance and stress testing device for computing nodes, please refer to the relevant description in the embodiment corresponding to the performance and stress testing method for computing nodes, which will not be repeated here.

[0230] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the performance and stress testing method for computing nodes.

[0231] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above embodiments of the performance and stress testing method for computing nodes.

[0232] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0233] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the embodiments of the performance and stress testing method for any of the computing nodes described above.

[0234] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the performance and stress testing methods for computing nodes.

[0235] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0236] The performance and stress testing method for a computing node and the electronic device provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A method for performance and stress testing of a computing node, characterized in that, Applied to an actuator, the method includes: Create test configuration files based on the test items, the corresponding test tools, and the test modes; wherein, the test modes include at least a single-component polling test mode and a multi-component parallel test mode; Obtain the mapping relationship between the topology of the compute nodes and the test configuration file; When it detects that it is connected to the computer device under test, it determines the target test configuration file corresponding to the topology of the computing nodes in the computer device under test according to the mapping relationship, and determines the target test item corresponding to the computing node in the computer device under test according to the test item selection operation on the user interface. Based on the target test configuration file, the target test tool and target test mode corresponding to the target test item are determined, and the target test tool and target test mode are sent to the computer device under test so that the target test tool can perform testing on the computer device under test. Obtain the test results of the target test tool on the target test item under the target test mode.

2. The performance and stress testing method for computing nodes according to claim 1, characterized in that, Before sending the target testing tool and the target testing mode to the computer device under test, the method further includes: The external communication connection method of the computing nodes is determined based on the topology of the computing nodes in the computer device under test; wherein, the external communication connection method includes at least dedicated interconnection method and general bus connection method; The test order of computing nodes in the computer device under test is determined based on the external communication connection method of the computing nodes. Sending the target testing tool and the target testing mode to the computer device under test so that the target testing tool can perform testing on the computer device under test includes: The target testing tool, the target testing mode, and the testing sequence are sent to the computer device under test, so that the target testing tool can test the target test items corresponding to the computing nodes in the computer device under test in accordance with the testing sequence and the target testing mode.

3. The performance and stress testing method for computing nodes according to claim 2, characterized in that, The method further includes: During the detection process of the target testing tool performing tests on the computer device under test, the topology information of the target computing node currently being tested is obtained; wherein, the topology information includes at least the correspondence between the target computing node and the non-unified memory access architecture node, and the communication link attributes between the target computing node and the non-unified memory access architecture node; Candidate communication paths are obtained based on the topology information of the target computing node; Obtain the transmission characteristics of candidate communication paths; wherein, the transmission characteristics include at least latency characteristics and transmission characteristics across nodes of a non-uniform memory access architecture; Based on the transmission characteristics of the candidate communication paths, the target candidate communication path with the minimum latency and which does not have the transmission characteristics of cross-node non-uniform memory access architecture is selected from the candidate communication paths. The status data of the computing node is obtained through the target candidate communication path; wherein, the status data includes at least core temperature, fan speed, power consumption, memory usage or error correction code count; If the detected status data is greater than the preset value corresponding to the status data, output a prompt message to characterize the test failure of the computing node.

4. The performance and stress testing method for computing nodes according to claim 3, characterized in that, After obtaining the status data of the computing node through the target candidate communication path, and before outputting a prompt message characterizing a test failure of the computing node when the status data is detected to be greater than a preset value corresponding to the status data, the method further includes: The acquired state data of the computing nodes is used as historical state data; Obtain the first historical state data within the first preset time period and obtain the prediction time step; Obtain the second historical state data within a preset time step, starting from the end of the first preset time. The first historical state data is used as the input sequence and the second historical state data is used as the output to train a deep learning model. The state data of the computing nodes within the preset time step before the time to be predicted is input into the trained deep learning model; The trained deep learning model outputs the state data of the computation nodes within the preset time step, starting from the time to be predicted.

5. The performance and stress testing method for computing nodes according to claim 3 or 4, characterized in that, Following the output of the prompt information characterizing the test failure of the computing node, the following is also included: The faulty computing node is determined based on the prompt information used to characterize the test failure of the computing node; Obtain the current test mode corresponding to the faulty computing node; Based on the current testing mode, different strategies are selected to migrate the corresponding test information when testing test items; wherein, the test information includes the corresponding runtime environment or test task during testing.

6. The performance and stress testing method for computing nodes according to claim 5, characterized in that, The migration of test information when selecting different strategies to test test items according to the current test mode includes: If the current test mode is detected to be a single compute node polling test mode, the locked memory page in the faulty compute node is obtained; wherein, the locked memory page records the current running environment corresponding to the test item during the test; Obtain the next computing node to be tested after the faulty computing node according to the testing order of the computing nodes in the computer device under test; The information in the locked memory page of the faulty compute node is migrated to the next compute node to be tested; so that the next compute node to be tested can determine the serial number of the compute node from which the information comes, and use the target testing tool to test the target test item corresponding to the next compute node to be tested; If the current test mode is detected to be a multi-compute node parallel test mode, the total number of compute nodes in the computer device under test and the number of compute nodes called are obtained; wherein, the compute nodes called are all compute nodes under test. If the number of computing nodes called is less than the total number of computing nodes, the test task corresponding to the test item of the faulty computing node is paused. Obtain the computing node to be migrated from the remaining computing nodes; wherein, the remaining computing nodes are the nodes in the computer device under test other than the computing node to be invoked; The test tasks corresponding to the test items of the faulty computing node are migrated to the computing node to be migrated, so that the target test items corresponding to the computing node to be migrated can be tested using the target test tool.

7. The performance and stress testing method for computing nodes according to claim 3 or 4, characterized in that, Following the output of the prompt information characterizing the test failure of the computing node, the following is also included: Obtain the current operating frequency of the faulty computing node; Adjust the current operating frequency to a preset operating frequency; wherein the preset operating frequency is less than the current operating frequency; If the operating frequency of the faulty computing node is detected to be at the preset operating frequency, the target test item corresponding to the computing node in the computer device under test is tested.

8. The method for testing the performance and stress of a computing node according to claim 1, characterized in that, The computer devices under test are computer devices with different operating systems; the connection between itself and the computer devices under test includes: its own unified interface to connect to different computer devices under test; After obtaining the test results of the target test item by the target test tool under the target test mode, the method further includes: The format of the test results should be determined based on the operating system of the computer device under test; The test results of the computer devices under test with different operating systems are converted into test results in the same format.

9. The performance and stress testing method for computing nodes according to claim 8, characterized in that, After obtaining the test results of the target test tool on the target test item under the target test mode, the method further includes: The test results are analyzed to obtain a report on the performance metrics and stability assessment results used to characterize the computing nodes; The operating parameters to be adjusted for the computing nodes are determined based on the reports of performance metrics and stability assessment results used to characterize the computing nodes.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the performance and stress testing method for a computing node as described in any one of claims 1 to 9 when executing the computer program.

Images