Intelligent computing cluster testing method, apparatus, device, medium, and computer program product

Abstract

The application discloses a kind of intelligent calculation cluster test method, device, equipment, medium and computer program product, the method includes: the initialization configuration of intelligent calculation cluster is carried out, each test link is executed in turn, and each test link is optimized in the testing process;Wherein, test link includes single machine comprehensive test, single machine model test, cluster communication test, cluster performance test and cluster long stable test;According to the target configuration of each test link after optimization, automatically arrange all test links, form test task sequence;In response to acceptance test request, execute test task sequence, and automatically generate acceptance test report according to the intermediate result of each test link.The application can more effectively locate problem server by integrating multiple incremental test links to test and optimize intelligent calculation cluster, thereby improving the accuracy and efficiency of fault location, and can arrange tasks, generate test report according to the latest test results, effectively improve test efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent computing cluster operation and maintenance technology, and in particular to an intelligent computing cluster testing method, apparatus, equipment, medium and computer program product. Background Technology

[0002] Computing power is the core productivity for training large-scale AI models. The success of large models like GPT-4 relies on the computing resources of tens of thousands of training GPUs. The explosion of large-scale models has directly led to a surge in computing power demand. Providing users with stable, standardized, and quantifiable GPU computing resources, saving users' costs and reducing the barriers to entry, and creating a safe and reliable server environment for long-term training and inference of user models is particularly important. AI cluster computing power has become the foundation of large-scale models. Therefore, verifying whether the various indicators of intelligent computing clusters meet delivery standards, and verifying the stability and high performance of intelligent computing clusters, have become urgent problems to be solved. However, at present, there is no complete and feasible testing method in the scenario of intelligent computing server optimization and acceptance testing, and the existing methods do not yet have the ability to be gradually optimized. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, device, medium, and computer program product for testing intelligent computing clusters. By integrating multiple incremental testing steps to test and optimize intelligent computing clusters, it can more effectively locate problematic servers, thereby improving the accuracy and efficiency of fault location. Furthermore, it can perform task orchestration and generate test reports with one click based on the latest test results, effectively improving testing efficiency.

[0004] To achieve the above objectives, embodiments of the present invention provide a method for testing intelligent computing clusters, including: The intelligent computing cluster is initialized and configured, and each test stage is executed sequentially, with optimization performed on each test stage during the testing process; the test stages include single-machine comprehensive testing, single-machine model testing, cluster communication testing, cluster performance testing, and cluster long-term stability testing. Based on the optimized target configuration of each test stage, all test stages are automatically arranged to form a test task sequence. In response to the acceptance test request, the test task sequence is executed, and an acceptance test report is automatically generated based on the intermediate results of each test stage.

[0005] As an improvement to the above solution, the single-machine comprehensive test includes: Check whether the version information of each node in the intelligent computing cluster meets the preset version requirements; wherein, the version information includes the operating system version, CPU kernel version, GPU card driver version, cluster communication framework version, and toolkit version; Check the presence status of the multi-GPU cards on each node in the intelligent computing cluster, verify the connectivity of the communication channels between the multi-GPU cards, and test whether the communication channel bandwidth meets the first preset threshold. Check the communication link status between the CPU and GPU card, and test whether the communication link bandwidth meets the second preset threshold.

[0006] As an improvement to the above solution, the single-machine model testing includes: The stand-alone model test materials based on the GPT3 model are distributed to each node in the intelligent computing cluster; wherein, the stand-alone model test materials include a stand-alone multi-GPU Docker image, training startup scripts, and training data; The model is trained using the test materials, and the average training time is calculated based on the training log after training is completed.

[0007] As an improvement to the above scheme, the cluster communication test includes: The communication performance of the intelligent computing cluster was tested to obtain the first test data; The intelligent computing cluster was subjected to a communication stability test to obtain the second test data; Based on the first test data and the second test data, the intelligent computing cluster is segmented; The communication performance test and the communication stability test were re-performed on the split intelligent computing cluster.

[0008] As an improvement to the above solution, the cluster performance test includes: The test materials for the multi-machine model built based on the LLama2-13B model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include a multi-machine multi-card Docker image, training startup scripts, and training data; Training is performed based on the multi-machine model test materials, and after training is completed, the average cluster computing power and the average effective computing power per card are calculated based on the training logs.

[0009] As an improvement to the above scheme, the long-term stability test of the cluster includes: The test materials for the multi-machine model built based on the GPT3 model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include multi-machine multi-card Docker images, training startup scripts, and training data; The training is performed on the multi-machine model test materials for a preset time, and the average cluster computing power and the average effective computing power of a single card are calculated based on the training logs after the training is completed.

[0010] This invention also provides a smart computing cluster testing device, comprising: The test and optimization module is used to initialize and configure the intelligent computing cluster, execute each test stage sequentially, and optimize each test stage during the test process; wherein, the test stage includes single-machine comprehensive test, single-machine model test, cluster communication test, cluster performance test, and cluster long-term stability test. The test orchestration module is used to automatically orchestrate all the test stages according to the optimized target configuration of each test stage, forming a test task sequence; The report generation module is used to respond to the acceptance test request, execute the test task sequence, and automatically generate an acceptance test report based on the intermediate results of each test stage.

[0011] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the intelligent computing cluster testing method described in any of the preceding embodiments.

[0012] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the intelligent computing cluster testing method described above.

[0013] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the intelligent computing cluster testing method described above.

[0014] Compared to existing technologies, the beneficial effects of the intelligent computing cluster testing method, apparatus, equipment, medium, and computer program product provided by this invention are as follows: By initializing and configuring the intelligent computing cluster, each test stage is executed sequentially, and each test stage is optimized during the testing process. The test stages include single-machine comprehensive testing, single-machine model testing, cluster communication testing, cluster performance testing, and cluster long-term stability testing. Based on the optimized target configuration of each test stage, all test stages are automatically arranged to form a test task sequence. In response to an acceptance test request, the test task sequence is executed, and an acceptance test report is automatically generated based on the intermediate results of each test stage. This invention integrates multiple incremental test stages to test and optimize the intelligent computing cluster, enabling more effective location of problematic servers, thereby improving the accuracy and efficiency of fault location. Furthermore, it allows for task orchestration and one-click generation of test reports based on the latest test results, effectively improving testing efficiency. Attached Figure Description

[0015] Figure 1This is a flowchart illustrating a preferred embodiment of a smart computing cluster testing method provided by the present invention; Figure 2 This is a schematic diagram of a preferred embodiment of the intelligent computing cluster testing device provided by the present invention; Figure 3 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating a preferred embodiment of a smart computing cluster testing method provided by the present invention. The smart computing cluster testing method includes: S1. Initialize and configure the intelligent computing cluster, execute each test step in sequence, and optimize each test step during the test; wherein, the test steps include single-machine comprehensive test, single-machine model test, cluster communication test, cluster performance test, and cluster long-term stability test. S2, based on the target configuration after optimization of each test stage, automatically arrange all the test stages to form a test task sequence; S3, in response to the acceptance test request, execute the test task sequence and automatically generate an acceptance test report based on the intermediate results of each test stage.

[0018] Specifically, this invention provides a method for testing intelligent computing clusters. First, the intelligent computing cluster is initialized and configured. Hardware integration with an LLD (Limited Layer Detailing) is used to automatically import and manage the intelligent computing server cluster, configure passwordless login between servers, install and configure various drivers and toolkits required for the intelligent computing server, and perform basic hardware checks and system environment configuration. For example, an LLD file can be imported to automatically generate an inventory.ini file, include the node IPs in the management network within the Ansible test management scope, and configure username / password login or keyless (public / private key) login. Depending on the actual situation, if plaintext password storage is allowed, username / password login can be used; otherwise, the ssh-copy-id command can be used to send the key to the destination host. After the cluster initialization and basic configuration are completed, connectivity testing is performed using Ansible's ping module. After the connectivity test is completed, each test stage is executed sequentially, and each test stage is optimized during the testing process. The test stages include single-machine comprehensive testing, single-machine model testing, cluster communication testing, cluster performance testing, and cluster long-term stability testing. It should be noted that the cluster acceptance scale increases progressively for each testing phase. Optimization testing during the sequential testing and acceptance process effectively locates errors throughout the project, improving the accuracy and efficiency of fault location, thereby ultimately achieving the acceptance and delivery of the entire intelligent computing cluster. Then, based on the optimized target configuration of each testing phase, all testing phases are automatically orchestrated using Ansible to form a test task sequence. Finally, in response to acceptance test requests, the test task sequence is executed, and an acceptance test report is automatically generated based on the intermediate results of each testing phase.

[0019] For example, embodiments of the present invention can perform batch connection tests of cluster nodes through the Ansible Playbook system, distribute test materials, execute test scripts, automatically collect test result logs, analyze and visualize the results based on the logs, and export the test results to an Excel spreadsheet. After the intelligent computing cluster test and acceptance is completed, a test report can be automatically generated based on the collected intermediate results. Embodiments of the present invention, combined with the above basic test process and other modules developed based on Ansible, can improve the work efficiency of tuning, testing, and acceptance personnel. After importing LLD, cluster servers are automatically managed, eliminating the need for manual maintenance of inventory files. Furthermore, it can perform complex operations such as batch orchestration and partitioning of the server cluster according to configuration requirements. After partitioning, it allows selection of whether to automatically enable passwordless login between smaller clusters, avoiding tedious manual passwordless login operations.

[0020] This invention divides the optimization and acceptance testing process into multiple steps and uses Ansible to implement task orchestration, enabling project-level intelligent computing server cluster optimization and acceptance testing capabilities. On one hand, this multi-stage acceptance testing method, by limiting different testing scopes, locates errors within a specific area, facilitating troubleshooting and problem-solving, and preserving intermediate results. On the other hand, after each stage of testing and acceptance is successfully completed, more convenient automatic task orchestration capabilities are achieved, and a final acceptance test report is generated.

[0021] In a preferred embodiment, the single-machine comprehensive test includes: Check whether the version information of each node in the intelligent computing cluster meets the preset version requirements; wherein, the version information includes the operating system version, CPU kernel version, GPU card driver version, cluster communication framework version, and toolkit version; Check the presence status of the multi-GPU cards on each node in the intelligent computing cluster, verify the connectivity of the communication channels between the multi-GPU cards, and test whether the communication channel bandwidth meets the first preset threshold. Check the communication link status between the CPU and GPU card, and test whether the communication link bandwidth meets the second preset threshold.

[0022] Specifically, in performing a single-machine comprehensive test, this embodiment of the invention checks whether the version information of each node in the intelligent computing cluster meets preset version requirements. This version information includes the operating system version, CPU kernel version, GPU card driver version, cluster communication framework version, and toolkit version. It also checks the presence status of the multiple GPU cards on each node in the intelligent computing cluster, verifies the connectivity of the communication channels between the multiple GPU cards, and tests whether the communication channel bandwidth meets a first preset threshold. Finally, it checks the communication link status between the CPU and GPU cards, executes allreduce and / or allgather operations, and tests whether the communication link bandwidth meets a second preset threshold.

[0023] For example, embodiments of the present invention can use Linux commands to obtain hardware information of each node in the intelligent computing cluster and output it to the test result log; use GPU bandwidth testing tools to test the replication bandwidth, data transmission and computing latency between devices, and append the test results to the test result log; use NCCL testing tools to check the performance and accuracy of NCCL operations, use the cuBLAS library to test the physical computing power of a single card, and append these results to the test result log; use Ansible-Playbook to collect the test result log, parse the required test items and export them to an Excel spreadsheet, and automatically mark the problematic test items.

[0024] In another preferred embodiment, the standalone model testing includes: The stand-alone model test materials based on the GPT3 model are distributed to each node in the intelligent computing cluster; wherein, the stand-alone model test materials include a stand-alone multi-GPU Docker image, training startup scripts, and training data; The model is trained using the test materials, and the average training time is calculated based on the training log after training is completed.

[0025] Specifically, in this embodiment of the invention, when performing single-machine model testing, Ansible-Playbook can be used to create directories on each node of the intelligent computing cluster, and the single-machine multi-GPU Docker image, training startup script, and training data, etc., built based on the open-source GPT3 model, can be distributed in batches via the management network. The Docker environment on each node of the intelligent computing cluster is checked for normal operation, and the existence of the GPT3 Docker image is checked. If it does not exist, the single-machine multi-GPU Docker image is imported using the Docker command; if a running GPT3 container exists, it is stopped and deleted. Then, the GPT3 container is started using the Docker command, the corresponding directory is mounted, and the training startup script is run for training. If the Docker command does not exist, the runtime environment is reconfigured in the current directory, and the above operations are performed. After training is completed, Ansible-Playbook is used to collect the single-machine model training logs of the intelligent computing cluster, and the training time is checked through scripts, error messages are output, the average training time and other intermediate results required for generating the report are calculated, and the results are visualized.

[0026] In yet another preferred embodiment, the cluster communication test includes: The communication performance of the intelligent computing cluster was tested to obtain the first test data; The intelligent computing cluster was subjected to a communication stability test to obtain the second test data; Based on the first test data and the second test data, the intelligent computing cluster is segmented; The communication performance test and communication stability test were re-performed on the split intelligent computing cluster.

[0027] Specifically, in this embodiment of the invention, cluster communication testing includes parameter plane set communication performance testing and parameter plane set communication stability testing. During cluster communication testing, the hostfile is automatically configured, declaring the IP addresses of the GPUs to be connected and specifying the number of GPUs on each node. Parameter plane set communication performance testing with 1G to 10G of data and parameter plane set communication stability testing with 10G to 10G of data are performed respectively. Data from both tests is collected, and the test results are analyzed to generate the intermediate results required for the report. When a communication test fails, the failed node can be removed, the intelligent computing cluster can be automatically split into small batches (powers of two), and batch passwordless logins can be performed within a small range. The communication performance and stability tests can then be restarted until the problematic server is located.

[0028] After successfully completing the cluster communication test, the embodiments of the present invention prove that the parameter plane communication is smooth and meets the conditions for cluster training performance testing and cluster training stability testing.

[0029] In yet another preferred embodiment, the cluster performance test includes: The test materials for the multi-machine model built based on the LLama2-13B model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include a multi-machine multi-card Docker image, training startup scripts, and training data; Training is performed based on the multi-machine model test materials, and after training is completed, the average cluster computing power and the average effective computing power per card are calculated based on the training logs.

[0030] Specifically, in this embodiment of the invention, the cluster training performance test is based on the LLaMA-13B open-source model, and the training parameters are adjusted to maximize the GPU's processing power (TFLOPS). Ansible-Playbook can be used to create directories on each node in the intelligent computing cluster, and multi-machine, multi-GPU Docker images, training startup scripts, and training data—all based on the open-source LLaMA-13B model—can be distributed in batches via the management network. Ansible-Playbook's capabilities are used for batch synchronization of small file scripts, specifically synchronizing the LLaMA-13B training startup script, node rank files, and dataset generation scripts. After distribution, the LLaMA-13B training startup script is executed, training logs are output to a specified log file, and training results are monitored. Based on the training results, the overall average computing power of the cluster and the average effective computing power per GPU are calculated and analyzed, and the statistical results are visualized, ultimately generating an intermediate report.

[0031] In yet another preferred embodiment, the cluster stability test includes: The test materials for the multi-machine model built based on the GPT3 model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include multi-machine multi-card Docker images, training startup scripts, and training data; The training is performed on the multi-machine model test materials for a preset time, and the average cluster computing power and the average effective computing power of a single card are calculated based on the training logs after the training is completed.

[0032] Specifically, in this embodiment of the invention, after completing the cluster training performance test, the optimal parameters for Nccl ensemble communication and the optimal training parameters for model training are obtained. The cluster training stability test is based on the GPT3 open-source model, and the GPT3 model training parameters are adjusted with reference to the optimal parameters from the cluster training performance test. Ansible-Playbook can be used to create directories on each node of the intelligent computing cluster, and multi-machine, multi-GPU Docker images, training startup scripts, and training data (all built based on the GPT3 model) can be distributed in batches via the management network. Ansible-Playbook's capabilities are used for batch synchronization of small file scripts, specifically synchronizing the GPT3 training startup script, the GPT3 distributed pre-training script, node rank files, and dataset generation scripts. After distribution, the GPT3 training startup script is executed for long-term training, such as 48 hours of continuous training. Training logs are output to a specified log file, and training results are monitored. Based on the training results, the overall average computing power of the cluster and the average effective computing power per GPU are calculated and analyzed, and the statistical results are visualized, ultimately generating an intermediate report.

[0033] After completing optimization and acceptance testing of all stages, all test stages can be automatically orchestrated into a task group, such as a test task sequence, based on the final optimized delivery form of each test stage, i.e., the target configuration of each test stage. According to the user's acceptance test request, the entire acceptance test process is automatically restarted in sequence, thus performing an end-to-end acceptance test delivery process. Intermediate results of each test stage are automatically saved, and a complete acceptance test report is ultimately generated.

[0034] Accordingly, the present invention also provides a smart computing cluster testing device, which can implement all the processes of the smart computing cluster testing method in the above embodiments.

[0035] Please see Figure 2 , Figure 2 This is a schematic diagram of a preferred embodiment of a smart computing cluster testing device provided by the present invention. The smart computing cluster testing device includes: The test optimization module 201 is used to initialize and configure the intelligent computing cluster, execute each test step in sequence, and optimize each test step during the test process; wherein, the test steps include single-machine comprehensive test, single-machine model test, cluster communication test, cluster performance test, and cluster long-term stability test. The test orchestration module 202 is used to automatically orchestrate all the test stages according to the target configuration after optimization of each test stage, forming a test task sequence; The report generation module 203 is used to respond to the acceptance test request, execute the test task sequence, and automatically generate an acceptance test report based on the intermediate results of each test stage.

[0036] Preferably, the single-machine comprehensive test includes: Check whether the version information of each node in the intelligent computing cluster meets the preset version requirements; wherein, the version information includes the operating system version, CPU kernel version, GPU card driver version, cluster communication framework version, and toolkit version; Check the presence status of the multi-GPU cards on each node in the intelligent computing cluster, verify the connectivity of the communication channels between the multi-GPU cards, and test whether the communication channel bandwidth meets the first preset threshold. Check the communication link status between the CPU and GPU card, and test whether the communication link bandwidth meets the second preset threshold.

[0037] Preferably, the single-machine model test includes: The stand-alone model test materials based on the GPT3 model are distributed to each node in the intelligent computing cluster; wherein, the stand-alone model test materials include a stand-alone multi-GPU Docker image, training startup scripts, and training data; The model is trained using the test materials, and the average training time is calculated based on the training log after training is completed.

[0038] Preferably, the cluster communication test includes: The communication performance of the intelligent computing cluster was tested to obtain the first test data; The intelligent computing cluster was subjected to a communication stability test to obtain the second test data; Based on the first test data and the second test data, the intelligent computing cluster is segmented; The communication performance test and the communication stability test were re-performed on the split intelligent computing cluster.

[0039] Preferably, the cluster performance test includes: The test materials for the multi-machine model built based on the LLama2-13B model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include a multi-machine multi-card Docker image, training startup scripts, and training data; Training is performed based on the multi-machine model test materials, and after training is completed, the average cluster computing power and the average effective computing power per card are calculated based on the training logs.

[0040] Preferably, the long-term stability test of the cluster includes: The test materials for the multi-machine model built based on the GPT3 model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include multi-machine multi-card Docker images, training startup scripts, and training data; The training is performed on the multi-machine model test materials for a preset time, and the average cluster computing power and the average effective computing power of a single card are calculated based on the training logs after the training is completed.

[0041] In specific implementation, the working principle, control process and technical effects of the intelligent computing cluster testing device provided in this embodiment of the invention are the same as those of the intelligent computing cluster testing method in the above embodiments, and will not be repeated here.

[0042] Please see Figure 3 , Figure 3 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 301, a memory 302, and a computer program stored in the memory 302 and configured to be executed by the processor 301. When the processor 301 executes the computer program, it implements the intelligent computing cluster testing method described in any of the above embodiments.

[0043] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 302 and executed by the processor 301 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0044] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 301 can be any conventional processor. The processor 301 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0045] The memory 302 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 302 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card, etc., or the memory 302 can also be other volatile solid-state storage devices.

[0046] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.

[0047] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the intelligent computing cluster testing method described in any of the above embodiments.

[0048] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the intelligent computing cluster testing method described in any of the above embodiments.

[0049] This invention provides a method, apparatus, device, medium, and computer program product for testing intelligent computing clusters. It initializes and configures the intelligent computing cluster, sequentially executes various test stages, and optimizes each stage during the testing process. The test stages include single-machine comprehensive testing, single-machine model testing, cluster communication testing, cluster performance testing, and cluster long-term stability testing. Based on the optimized target configuration of each test stage, all test stages are automatically arranged to form a test task sequence. In response to an acceptance test request, the test task sequence is executed, and an acceptance test report is automatically generated based on the intermediate results of each test stage. This invention integrates multiple incremental test stages for testing and optimizing the intelligent computing cluster, enabling more effective location of problematic servers, thereby improving the accuracy and efficiency of fault location. Furthermore, it allows for task orchestration and one-click generation of test reports based on the latest test results, effectively improving testing efficiency.

[0050] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0051] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for testing intelligent computing clusters, characterized in that, include: The intelligent computing cluster is initialized and configured, and each test stage is executed sequentially, with optimization performed on each test stage during the testing process; the test stages include single-machine comprehensive testing, single-machine model testing, cluster communication testing, cluster performance testing, and cluster long-term stability testing. Based on the optimized target configuration of each test stage, all test stages are automatically arranged to form a test task sequence. In response to the acceptance test request, the test task sequence is executed, and an acceptance test report is automatically generated based on the intermediate results of each test stage.

2. The intelligent computing cluster testing method as described in claim 1, characterized in that, The single-machine comprehensive test includes: Check whether the version information of each node in the intelligent computing cluster meets the preset version requirements; wherein, the version information includes the operating system version, CPU kernel version, GPU card driver version, cluster communication framework version, and toolkit version; Check the presence status of the multi-GPU cards on each node in the intelligent computing cluster, verify the connectivity of the communication channels between the multi-GPU cards, and test whether the communication channel bandwidth meets the first preset threshold. Check the communication link status between the CPU and GPU card, and test whether the communication link bandwidth meets the second preset threshold.

3. The intelligent computing cluster testing method as described in claim 1, characterized in that, The single-machine model testing includes: The stand-alone model test materials based on the GPT3 model are distributed to each node in the intelligent computing cluster; wherein, the stand-alone model test materials include a stand-alone multi-GPU Docker image, training startup scripts, and training data; The model is trained using the test materials, and the average training time is calculated based on the training log after training is completed.

4. The intelligent computing cluster testing method as described in claim 1, characterized in that, The cluster communication test includes: The communication performance of the intelligent computing cluster was tested to obtain the first test data; The intelligent computing cluster was subjected to a communication stability test to obtain the second test data; Based on the first test data and the second test data, the intelligent computing cluster is segmented; The communication performance test and communication stability test were re-performed on the split intelligent computing cluster.

5. The intelligent computing cluster testing method as described in claim 1, characterized in that, The cluster performance test includes: The test materials for the multi-machine model built based on the LLama2-13B model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include a multi-machine multi-card Docker image, training startup scripts, and training data; Training is performed based on the multi-machine model test materials, and after training is completed, the average cluster computing power and the average effective computing power per card are calculated based on the training logs.

6. The intelligent computing cluster testing method as described in claim 1, characterized in that, The cluster stability test includes: The test materials for the multi-machine model built based on the GPT3 model are distributed to the intelligent computing cluster; wherein, the test materials for the multi-machine model include multi-machine multi-card Docker images, training startup scripts, and training data; The training is performed on the multi-machine model test materials for a preset time, and the average cluster computing power and the average effective computing power of a single card are calculated based on the training logs after the training is completed.

7. A smart computing cluster testing device, characterized in that, include: The test and optimization module is used to initialize and configure the intelligent computing cluster, execute each test stage sequentially, and optimize each test stage during the test process; wherein, the test stage includes single-machine comprehensive test, single-machine model test, cluster communication test, cluster performance test, and cluster long-term stability test. The test orchestration module is used to automatically orchestrate all the test stages according to the optimized target configuration of each test stage, forming a test task sequence; The report generation module is used to respond to the acceptance test request, execute the test task sequence, and automatically generate an acceptance test report based on the intermediate results of each test stage.

8. A terminal device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor executes the computer program to implement the intelligent computing cluster testing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the intelligent computing cluster testing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the intelligent computing cluster testing method as described in any one of claims 1 to 6.

Images