Automated testing system and method for direct communication between heterogeneous GPUs

Abstract

The present invention discloses an automated testing system and method for direct communication between heterogeneous GPU machines. The method includes: a user checks and configures test parameters and submits a test request, a test manager parses the request and performs scheduling processing, executes a specific test process, and finally collects test data, and performs result analysis and test report generation. To implement the above method, the system includes the following modules: a user interface module, a test manager module, a configuration database module, a test executor module, a network interface module, a result analyzer module, and a report generator module. The modules coordinate and process to complete the automated testing of high-speed communication between machines in a heterogeneous GPU environment under the GPU Direct RDMA mode. The system supports the software and hardware configurations of multiple GPU manufacturers, provides standardized test processes and methods, and solves the problems of cumbersome and inefficient GDR testing in a heterogeneous GPU environment.

Description

Technical Field

[0001] The present invention relates to the field of software testing technology, specifically to the field of testing for high-performance computing scenarios, and more particularly to an automated testing system and method for direct communication between heterogeneous GPU machines. Background Art

[0002] Large model training is an important research direction in the field of artificial intelligence, which involves building and optimizing neural network models with large parameter magnitudes. Large models and large-scale convolutional neural networks can show excellent performance in many fields such as image recognition, natural language processing, and speech recognition. As a processor specially designed for graphics rendering, GPU has a large number of computing cores and is also an ideal choice for deep learning and other high-performance computing tasks with its parallel processing capabilities. In model training and other high-performance computing scenarios, communication between GPUs on different nodes is an indispensable link. GPU Remote Direct Memory Access (GPU Direct RDMA, GDR) allows GPUs and third-party devices using the PCIE standard to exchange data directly without involving the CPU. Traditionally, when data needs to be transferred between the GPU and another device, the data must pass through the CPU, resulting in potential bottlenecks and increased latency. With GPU Direct RDMA, network adapters and storage drives can directly read and write GPU memory, reducing unnecessary memory consumption, reducing CPU overhead and reducing latency, thereby significantly improving performance. Currently, network communication has become a performance bottleneck for distributed machine learning, and GDR technology is crucial to improving the performance of GPU machine-to-machine communication.

[0003] However, the existing technology has many limitations in testing GDR communication in a heterogeneous GPU environment. First, the environmental complexity of the test process is high. Since the heterogeneous GPU environment involves a variety of hardware architectures and software stacks, there are many factors to consider when conducting GDR communication testing, such as device compatibility, driver configuration, network topology, etc., which makes the test process cumbersome and difficult to manage. Secondly, the testing efficiency of the existing technology is low. Due to the lack of a unified testing framework and automated tools, and the large scale of distributed clusters, the entire test cycle is long, and it is difficult to quickly locate communication problems, which affects the environmental detection and performance optimization of the entire system.

[0004] In addition, too much manual intervention is also a major limitation of existing technologies. Professionals are required to adjust and optimize various parameters during the test process, which not only increases labor costs, but also easily leads to errors in test results due to human factors. Therefore, in order to overcome these limitations, a more efficient, automated and easy-to-manage GDR communication test method for heterogeneous GPU machines is urgently needed. Through automated testing, the performance of heterogeneous GDR systems under different configurations can be quickly and accurately evaluated, and problems can be discovered and solved in a timely manner. Summary of the invention

[0005] In view of the deficiencies in the prior art, an object of the present invention is to provide an automated testing system and method for direct communication between heterogeneous GPUs.

[0006] The objective of the present invention is achieved through the following technical solution: an automated testing method for direct communication between heterogeneous GPUs, comprising the following steps:

[0007] Enter the test request and define the test parameters; the test parameters include the GPU model of the machine in the heterogeneous cluster, network configuration parameters, and the GPU RUNTIME memory management related function name and header file path;

[0008] Receive and parse the test request and test parameters, detect and update relevant parameters to the configuration database in real time according to the content of the test request, and the relevant parameters include test cases, machine equipment information, and network parameters;

[0009] Based on the latest test configuration retrieved from the configuration database, the network interface module is called for preprocessing to complete the software stack environment and network environment detection; GPU Direct RDMA performance test cases are executed between GPU devices of specified heterogeneous or homogeneous machines to ensure that the test is carried out in accordance with the predetermined test parameters;

[0010] During the test execution, the performance test data of GPU Direct RDMA is collected in real time; the performance test data is processed and analyzed, and the key information of the predicted test results stored in the configuration database is compared with the performance test data to determine whether the test meets the expected performance standards; key information such as the maximum bandwidth measured in the performance data should reach 95% of the theoretical rate of the network card;

[0011] Based on the analysis results of the result analyzer, a detailed test report is created, which includes a test summary, performance data, and result comparison.

[0012] Furthermore, the inputting of the test request and defining the test parameters includes: selecting different test scenarios, wherein the test scenarios include a single transmission test, a continuous transmission test or a stress test.

[0013] Furthermore, the parsing of the test request and the test parameters includes: parsing input parameters of a user interface, updating a configuration database module, querying data in a configuration database, starting a test executor, and obtaining a test report after the test is completed.

[0014] Furthermore, after the preprocessing, the GDR test code is automatically generated, such as injecting the memory management code of the specified GPU based on the open source general Perftest warehouse code, and injecting the compilation conditions to generate an executable program.

[0015] Furthermore, the performance test data includes key performance indicators such as transmission rate, delay, and bandwidth.

[0016] The present invention also provides an automated testing system for direct communication between heterogeneous GPUs, the system comprising the following modules:

[0017] User interface module: used to receive user configuration parameters, issue parameter configuration and test command requests;

[0018] Test manager component module: used to parse user interface input, read and write configuration database modules, start test executors and query result reports;

[0019] Configuration database module: used to store GPU memory management information and network parameter information of heterogeneous GDR test machines;

[0020] Test executor module: used to schedule network interface preprocessing, automatically generate GDR test code, start performance testing and collect test data in real time;

[0021] Network interface module: used for software stack and driver detection, IB adapter discovery, link check and network configuration verification;

[0022] Result Analyzer Module: used for threshold comparison, anomaly detection and performance analysis of test result data;

[0023] Report generator module: used to collect statistics on abnormal information, compare data and performance data, and generate complete reports.

[0024] Furthermore, the test manager component module is also used to read and write configuration database modules, start test executors and query result reports.

[0025] Furthermore, the configuration database module is also used to store test cases, GPU device RUNTIME information of heterogeneous machines, RDMA network parameter configuration information and key information of expected test results.

[0026] Furthermore, the test executor module supports multiple heterogeneous GPU Direct RDMA test scenarios and has real-time detection and exception handling mechanisms.

[0027] Furthermore, the complete report generated by the report generator module includes a test summary, performance data, and problem diagnosis.

[0028] The beneficial effects of the present invention are: by optimizing the test process and automated management, the efficiency and accuracy of heterogeneous GDR testing are significantly improved. Manual intervention is reduced, the complexity and error rate of the test process are reduced, and the cluster establishment time is accelerated. At the same time, the method can more accurately diagnose and locate the system's high network communication performance bottleneck through detailed test steps and real-time data collection, providing strong technical support for the optimization of high-performance computing systems. In addition, the present invention helps to improve the network communication performance in the data center and supercomputing fields, reduce operating costs, and provide guarantees for the reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0030] Figure 1 A schematic diagram of a heterogeneous GDR communication system according to the present invention;

[0031] Figure 2 It is a diagram for dividing the automated test modules of the GDR communication between heterogeneous GPUs described in the present invention;

[0032] Figure 3 This is a UI diagram of the automated test user interface for GDR communication between heterogeneous GPUs according to the present invention;

[0033] Figure 4 This is a flow chart of automated testing of GDR communication between heterogeneous GPUs according to the present invention;

[0034] Figure 5 This is a flow chart of a network interface preprocessing module in the test executor of the present invention;

[0035] Figure 6 The present invention provides a flow chart for automatically generating GDR test codes and executable programs in the test executor described in the present invention. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0037] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0039] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0040] Embodiment 1

[0041] like Figure 1 As shown, the heterogeneous GDR communication system described in the present invention, GPUs of different manufacturers (A, B, C, D) are connected together through GPU Direct RDMA technology. According to different subnet divisions and routing designs, each combination has the communication capability of receiving and sending high-speed network data.

[0042] In the field of high-performance computing and deep learning, as model complexity and data volume increase, a single GPU can no longer meet the demand, and multiple GPUs or even multiple servers working together has become the norm. In order to improve the efficiency of data transmission and reduce latency, heterogeneous GDR (GPU Direct RDMA) technology came into being. GDR technology allows direct data transmission and communication between GPUs of different types or generations, and between GPUs and RDMA network devices without the involvement of the CPU, thereby significantly improving system performance.

[0043] The core of heterogeneous GPU Direct RDMA is to use RDMA (Remote Direct Memory Access) technology to achieve direct memory access between GPUs or between GPUs and network devices. RDMA technology moves data directly between applications through hardware, bypassing the operating system protocol stack, significantly improving network communication efficiency. This technology greatly improves the efficiency and performance of network communication and is suitable for scenarios such as high-performance computing, big data processing, and distributed storage. It has the following characteristics: Zero Copy: By eliminating the copying of data between kernel space and user space, transmission latency is significantly reduced; Kernel Bypass: Applications can send commands directly to the network card without executing kernel memory calls, reducing the number of environment switches, thereby greatly accelerating the rate of GPU data interaction between machines.

[0044] Embodiment 2

[0045] like Figure 2 As shown, an automatic test system for direct communication between heterogeneous GPUs is also provided in an embodiment of the present invention. The system includes the following modules:

[0046] User interface module: provides an interactive platform for users to create and manage test plans, input and update test parameters, configure node information, select different test scenarios such as single transmission test, continuous transmission test or stress test, start and stop test commands, and download and view test reports. This module has a user-friendly interface and is easy to operate, supporting users to access and control the automated test system.

[0047] Test manager module: This module parses the input commands of the user interface, manages and configures network information, reads and writes configuration databases, starts test executors and queries result reports. This module has a task scheduling algorithm to ensure that test tasks are executed in order according to preset priorities and dependencies; at the same time, it can dynamically adjust test configurations to adapt to different test scenarios.

[0048] Configuration database module: This module uses a relational database management system to store test configuration information, such as the IP address, port number, test data size, and machine network information of heterogeneous environment machines. It uses a relational model to directly and intuitively display data in a table. The designed configuration database includes tables, fields, and relationships to store test configurations and results. The database design of this module supports fast retrieval and update to ensure the accuracy and real-time nature of the test configuration.

[0049] Network interface module: This module performs software and hardware environment detection: the software environment includes the operating system, driver, test tools, etc.; the hardware environment includes IB adapter discovery and link status check under the organization of GPU, network card, network switch, etc., as well as verification of network configuration parameters. This module supports automatic network configuration and fault detection, and optimizes network communication performance.

[0050] Test Executor Module: This module calls the network interface module for preprocessing, automatically generates GDR test code, executes test cases according to the instructions of the test manager module, and sends the test results to the result analyzer module. This module supports multiple heterogeneous GPU Direct RDMA test scenarios and has real-time detection and exception handling mechanisms.

[0051] Result Analyzer Module: This module analyzes the test results sent by the Test Executor Module, calls the threshold comparison unit, anomaly detection unit, and performance analysis unit to determine whether the test passes. This module uses an efficient data analysis algorithm to quickly locate performance bottlenecks.

[0052] Report Generator Module: This module generates a detailed test report based on the test result summary of the Result Analyzer Module, including test summary, performance data, and problem diagnosis information. This module supports multiple report formats and provides customized report generation capabilities.

[0053] Embodiment 3

[0054] like Figure 3 As shown, it is a user interface UI diagram of the system of the present invention, which is divided into three parts: machine configuration parameters, test case parameters, and key instructions.

[0055] The machine configuration parameters support retrieving the existing configuration in the database through the management network IP, or adding the machine configuration of the heterogeneous cluster on the Tab page, configuring the GPU manufacturer's RUNTIME memory management function, network MTU configuration, OP number configuration, etc. for each machine, and updating the configuration database after the configuration is sent to the machine.

[0056] The test case parameters configure the node execution scope of this automated test, including point-to-point testing and database permutation and combination testing. The point-to-point test is used to quickly test and verify the performance of two machines in the cluster using GDR, while the permutation and combination method scans and performs comprehensive connectivity and performance tests on all cluster machines in the test node group. Users can choose different test scenarios, such as single transmission test, continuous transmission test, or stress test.

[0057] Key instructions include starting a test, stopping a test, obtaining a test report, querying parameters, and updating a database. The test manager receives parameters and key instructions.

[0058] Embodiment 4

[0059] like Figure 4 As shown, an automated testing method for direct communication between heterogeneous GPUs is provided in an embodiment of the present invention, comprising the following steps:

[0060] Step 1: Submit a test request

[0061] The user enters the test request through the graphical user interface (GUI), defines the test parameters, and starts the test command. These parameters include but are not limited to the management IP address of the heterogeneous node, GPU model, network configuration, data transmission size, data transmission mode, etc. The user can select different test scenarios, such as single transmission test, continuous transmission test or stress test, and issue a start test command.

[0062] Step 2: Test scheduling process

[0063] As the central scheduling unit of the system, the test manager receives test requests submitted by users, retrieves information in real time, and updates the corresponding test configuration information from the configuration database according to the request content. The test configuration information includes test cases, device information, network parameters, etc. The test manager is also responsible for allocating test resources, such as GPU devices, network bandwidth, and storage resources, and ensuring the consistency and isolation of the test environment. Then it schedules the test executor to work.

[0064] Step 3: Test execution process

[0065] The test executor calls the network interface module for preprocessing and detects the hardware and software environment based on the test configuration retrieved from the configuration database. Then, the test executor automatically generates the GDR test code, that is, based on the general perftest warehouse code, injects the memory management code and compilation conditions of the specified GPU to generate an executable program. The GPU Direct RDMA performance test case is executed on the specified GPU device to ensure that it is carried out according to the predetermined test parameters and collects test data during the test process.

[0066] Step 4: Performance data collection

[0067] During the test execution, the test executor collects GPU Direct RDMA performance test data in real time. This data includes key performance indicators such as transmission rate, latency, bandwidth utilization, and error rate. The test executor also records system logs and error information during the test process for subsequent analysis.

[0068] Step 5: Result analysis and processing

[0069] The Result Analyzer receives the performance data collected by the test executor and processes and analyzes it. It compares the actual test results with the expected results calculated by storing key information in the configuration database to determine whether the test meets the performance standards. In addition, the Result Analyzer identifies performance bottlenecks, abnormal data, and potential hardware or software problems.

[0070] Step 6: Generate test report

[0071] The report generator creates a detailed test report based on the analysis results of the result analyzer. The report content usually includes test summary, performance data, result comparison, potential problem analysis, optimization suggestions, etc. The report can be output in multiple formats, such as PDF, HTML or Excel, for users to view and share.

[0072] Embodiment 5

[0073] like Figure 5 As shown, the network interface preprocessing module flow of the key step of the test executor in the present invention is performed. In the software stack and driver detection step, RDMA-CORE user-mode environment detection and RDMA IBVerbs kernel driver detection are performed according to the operating system type and version; detection of dependent user-mode software, such as libibverbs-dev, librdmacm-dev, libpci-dev, and GPU RUNTIME driver detection are performed. In the IB adapter discovery step, the key tools ibv_devinfo, ibstat, ibstatus, and show_gids of RDMA-CORE are called to check the status of the IB adapter. The database is retrieved to obtain the gateway information of the machine, the routing table is detected, and the basic ping packet test between nodes is performed to confirm that the connection is reachable. The detection confirms the status of the GPU device, checks the availability of the network connection, and verifies whether the required libraries and dependencies have been installed.

[0074] Embodiment 6

[0075] like Figure 6 FIG. 1 is a flow chart of automatically generating GDR test code and executable program in the test executor of the present invention.

[0076] Before automatically generating GDR test code, follow Figure 5 The pre-processing module first checks the test environment to ensure that all necessary hardware and software resources are ready.

[0077] Based on the detection of the preprocessing module, the existence of the runtime.h file is determined. If the file does not exist, the test executor will report the error. If the file exists, code injection is performed. According to the file, the memory management interface function code is automatically generated, and the memory allocation and release code of the corresponding GPU are injected into the general RDMA Perftest test code and its compilation option configuration; the test executor calls a pre-written Bash script, which is responsible for compiling the test code and generating an executable program. The script automatically handles all steps in the compilation process, including preprocessing, compilation, linking, and post-processing.

[0078] After compilation, the test executor generates two main executable programs: one for testing bandwidth performance and the other for testing latency performance.

[0079] These executables are designed to run on specific GPU devices to evaluate the performance of GPU Direct RDMA.

[0080] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0081] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0083] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0084] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An automated testing method for direct communication between heterogeneous GPUs, characterized in that: The following steps are involved: Enter the test request and define the test parameters; the test parameters include the GPU model of the machine in the heterogeneous cluster, network configuration parameters, and the GPU RUNTIME memory management related function name and header file path; Parse the test request and test parameters, detect and update relevant parameters to the configuration database in real time according to the content of the test request, and the relevant parameters include test cases, machine equipment information, and network parameters; Determine the test environment according to the latest test configuration retrieved from the configuration database, and perform test environment preprocessing, wherein the preprocessing includes software stack environment and network environment detection; execute GPUDirect RDMA performance test cases between GPU devices of specified heterogeneous or homogeneous machines; During the test execution, the performance test data of GPU Direct RDMA is collected in real time; Processing and analyzing the performance test data, and comparing the performance test data with the predicted test result key information stored in the configuration database to determine whether the test meets the expected performance standard; The test data is analyzed to obtain detailed analysis results and a test report is created, wherein the test report includes a test summary, performance data, and result comparison.

2. The automated testing method for direct communication between heterogeneous GPUs according to claim 1, characterized in that: The input test request and the definition of test parameters include: selecting different test scenarios, wherein the test scenarios include a single transmission test, a continuous transmission test or a stress test.

3. The automated testing method for direct communication between heterogeneous GPUs according to claim 1, characterized in that: The parsing of the test request and the test parameters includes: parsing input parameters of the user interface, updating the configuration database module, querying data in the configuration database, starting the test executor, and obtaining a test report after the test is completed.

4. The automated testing method for direct communication between heterogeneous GPUs according to claim 1, characterized in that: After the preprocessing, a GPU Direct RDMA test code is generated.

5. The automated testing method for direct communication between heterogeneous GPUs according to claim 1, characterized in that: The performance test data includes transmission rate, delay, and bandwidth.

6. An automated testing system for direct communication between heterogeneous GPUs, characterized in that: Includes the following modules: User interface module: used to receive and send user test requests and test parameters; Test manager component module: used to parse the test request and test parameters, detect and update relevant parameters to the configuration database in real time according to the content of the test request; Configuration database module: used to store GPU memory management information and network parameter information of heterogeneous GPU Direct RDMA test machines; Test executor module: used to schedule network interface preprocessing, automatically generate GPU Direct RDMA test code, start performance testing and collect test data in real time; Network interface module: used for software stack and driver detection, IB adapter discovery, link check and network configuration verification; Result Analyzer Module: used for threshold comparison, anomaly detection and performance analysis of test result data; Report generator module: used to collect statistics on abnormal information, compare data and performance data, and generate complete reports.

7. The automated testing system for direct communication between heterogeneous GPUs according to claim 6, characterized in that: The test manager component module is also used to read and write configuration database modules, start test executors and query result reports.

8. The automated testing system for direct communication between heterogeneous GPUs according to claim 6, characterized in that: The configuration database module is also used to store test cases, GPU device RUNTIME information of heterogeneous machines, RDMA network parameter configuration information and key information of expected test results.

9. The automated testing system for direct communication between heterogeneous GPUs according to claim 6, characterized in that: The test executor module supports multiple heterogeneous GPU Direct RDMA test scenarios and has real-time detection and exception handling mechanisms.

10. The automated testing system for direct communication between heterogeneous GPUs according to claim 6, characterized in that: The complete report generated by the report generator module includes test summary, performance data, and problem diagnosis.

Images