Communication testing methods, apparatus, equipment, storage media and program products

Abstract

This application discloses a communication testing method, apparatus, device, storage medium, and program product. The method includes: constructing a test environment comprising multiple nodes to be tested; performing pre-tests on each of the nodes to be tested within the test environment, the pre-tests being used to check the network connectivity and resource availability of the nodes to be tested; and, if the pre-tests pass, performing multi-node communication tests on the multiple nodes to be tested to obtain performance data, the performance data being used to characterize the communication performance between the multiple nodes to be tested. By executing communication tests with complete test processing logic, testers are not required to manually configure test parameters or execute test procedures, thus improving the efficiency of communication testing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a communication testing method, apparatus, device, storage medium, and program product. Background Technology

[0002] A multi-node machine learning platform cluster refers to an integrated computing environment consisting of multiple independent computing nodes interconnected via a network to work collaboratively to support large-scale machine learning tasks. These large-scale machine learning tasks include, but are not limited to, distributed computing and model inference.

[0003] In related technologies, when conducting communication tests on the aforementioned multi-node machine learning platform cluster, testers need to manually check the network environment and resource information of each node, configure the test environment for each node, and start the communication test program on each node based on the manually set test parameters. After the test is completed, the test results of each node need to be collected and processed manually.

[0004] However, the manual operation in the above communication test process is cumbersome and time-consuming, especially in scenarios with a large number of nodes. The efficiency of performing configuration processing on each node is low, resulting in low efficiency of communication testing for multi-node machine learning platform clusters. Summary of the Invention

[0005] This application provides a communication testing method, apparatus, device, storage medium, and program product. The technical solution is as follows.

[0006] On the one hand, a communication testing method is provided, the method comprising: Build a test environment, which includes multiple nodes to be tested; In the test environment, pre-tests are performed on the plurality of nodes to be tested, and the pre-tests are used to check the network connectivity and resource availability of the nodes to be tested. If the pre-test passes, a multi-node communication test is performed on the plurality of nodes to be tested to obtain performance data, which is used to characterize the communication performance between the plurality of nodes to be tested.

[0007] On the other hand, a communication testing apparatus is provided, the apparatus comprising: The environment building module is configured to build a test environment, which includes multiple nodes to be tested. The pre-test module is configured to perform pre-tests on the plurality of nodes to be tested in the test environment, wherein the pre-tests are used to check the network connectivity and resource availability of the nodes to be tested; The communication testing module is configured to perform multi-node communication tests on the plurality of nodes to be tested if the pre-test passes, and obtain performance data, which is used to characterize the communication performance between the plurality of nodes to be tested.

[0008] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the communication testing method as described in any of the embodiments of this application above.

[0009] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the communication testing method as described in any of the embodiments of this application above.

[0010] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the communication testing methods described in the above embodiments.

[0011] The beneficial effects of the technical solutions provided in this application include at least the following: By constructing a unified test environment, configuration processing is performed on multiple nodes under test separately. Pre-tests and multi-node communication tests are automatically executed sequentially within the test environment. The pre-test serves as a preliminary check for multi-node communication tests. Once the pre-test passes, meaning the network connectivity and resources of the node under test are available, the communication test is executed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing redundant debugging costs, lowering manpower costs, and reducing configuration time, thereby improving the efficiency of communication testing. Furthermore, the communication problems on a single node under test indicated by the pre-test are decoupled from the communication problems between at least two nodes under test indicated by the multi-node communication test, enabling the location of communication anomalies to a single node under test and improving the accuracy of communication testing. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a communication test system provided in an exemplary embodiment of this application; Figure 2 This is a flowchart of a communication testing method provided in an exemplary embodiment of this application; Figure 3 This is a flowchart of a communication testing method provided in another exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the process of creating a container instance provided in an exemplary embodiment of this application; Figure 5 This is a flowchart of a communication testing method provided in yet another exemplary embodiment of this application; Figure 6 This is a schematic diagram of the overall communication testing process provided in an exemplary embodiment of this application; Figure 7 This is a flowchart of a communication testing method provided in another exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of a test system execution module provided in an exemplary embodiment of this application; Figure 9 This is a structural block diagram of a communication testing apparatus provided in an exemplary embodiment of this application; Figure 10 This is a structural block diagram of a communication testing apparatus provided in another exemplary embodiment of this application; Figure 11 This is a structural block diagram of a terminal provided in an exemplary embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0016] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0018] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user-related data (e.g., performance data involved in this application). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their relevant data is being collected. This ensures that the application only begins the steps for acquiring user-related data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without confirmation from the user), the steps for acquiring user-related data end, meaning no user-related data is acquired. In other words, all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, operational data and account information involved in this application are obtained with full authorization.

[0019] A multi-node machine learning platform cluster refers to an integrated computing environment consisting of multiple independent computing nodes interconnected via a network to work collaboratively to support large-scale machine learning tasks. These large-scale machine learning tasks include, but are not limited to, distributed computing and model inference. The communication performance within a multi-node machine learning platform cluster directly affects the cluster's computational efficiency, and thus its ability to execute large-scale machine learning tasks.

[0020] In related technologies, to test the communication performance within a multi-node machine learning platform cluster, communication tests are performed on multiple nodes under test within the cluster. That is, performance data indicating cluster communication performance is obtained through data exchange operations between multiple nodes under test. Communication testing methods include, but are not limited to, at least one of the following.

[0021] 1. Testers need to manually check the network environment and resource information of each node under test, configure the test environment for each node under test, and start the communication test program on each node under test based on the manually set test parameters to perform the above communication test; after the test is completed, the test results of each node under test need to be collected and processed manually.

[0022] Configuring the test environment for each node under test involves steps such as deploying test programs, installing and running dependencies, and creating and mounting shared storage environments. All of these steps require manual configuration and verification by testers. However, manual testing lacks comprehensive verification of the runtime environment of each node under test, including hardware status verification, driver status verification, and software stack integrity verification. In cases where internal link anomalies, driver anomalies, or inconsistent software environments exist on the nodes under test, communication tests between multiple nodes may fail or produce abnormal results.

[0023] 2. Based on the communication testing process, testers create modular scripts that encapsulate at least one step of node login, test environment configuration, test program startup, and test result collection and processing, enabling batch execution on each node under test. After the test is completed, the test results of each node under test still need to be collected and processed manually.

[0024] The script used to configure the test environment includes test node information, test parameters for starting the communication test program, and running path. These configuration parameters are set based on the specific configuration parameters of the node to be tested. That is, if the number or information of the nodes to be tested or the network topology between multiple nodes to be tested changes, the content of the previously created script needs to be modified. Furthermore, the test environment for each node needs to be confirmed by the tester.

[0025] The aforementioned communication testing methods suffer from at least the following problems: 1. Configuration information and test results in the testing process are scattered across various test nodes and test stages, making manual processing cumbersome and time-consuming. Even script-based testing methods still require manual modification of script content when the test environment changes, resulting in high complexity. This is especially true in scenarios with a large number of nodes, where the efficiency of configuration processing for each node is low, leading to low efficiency in communication testing of multi-node machine learning platform clusters; 2. When a single node experiences a communication problem, the test program runs in multi-node communication mode. Consequently, it only reflects abnormal multi-node communication test results and cannot pinpoint internal runtime environment issues such as resource unavailability or test program compilation failures on a single node. Therefore, it cannot obtain true performance data for multi-node communication, resulting in low stability and low accuracy of the testing process.

[0026] The communication testing method provided in this application constructs a unified test environment to perform configuration processing on multiple nodes under test. Pre-tests and multi-node communication tests are automatically executed sequentially within the test environment. The pre-test serves as a preliminary check for multi-node communication testing. Once the pre-test passes (i.e., the network of the node under test is connected and resources are available), the communication test is executed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing redundant debugging costs, lowering labor costs and configuration time, thereby improving communication testing efficiency. Furthermore, it decouples communication problems on a single node under test indicated by the pre-test from communication problems between at least two nodes under test indicated by the multi-node communication test, enabling the location of communication anomalies to a single node under test and improving the accuracy of communication testing.

[0027] First, the communication testing system of this application will be introduced. Please refer to... Figure 1 The illustration shows a schematic diagram of a communication test system provided in an exemplary embodiment of this application, which includes a computer device 10.

[0028] Computer device 10 is used to automate the communication testing process for multiple nodes under test. The nodes under test can be heterogeneous machine learning platforms, which are hardware and software system architectures that integrate various types of computing units to collaboratively execute machine learning tasks. These platforms schedule the tasks to corresponding hardware processors based on the characteristics of the machine learning tasks. Each node under test includes at least one hardware processor, including but not limited to graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), and application-specific integrated circuits (ASICs). Optionally, the nodes under test may include at least one terminal or server, etc., but this embodiment does not limit the scope of the application.

[0029] The computer device 10 can be a device within a node to be tested, meaning it can be implemented as a node to be tested in a test environment; alternatively, the computer device 10 can be a device connected to a machine learning platform cluster corresponding to multiple nodes to be tested, and this embodiment does not limit this. Indicatively, the computer device 10 can be implemented as a control device for controlling a machine learning platform cluster corresponding to multiple nodes to be tested. The computer device 10 is connected to multiple nodes to be tested respectively and is used to perform coordination and control processing such as test task distribution, test timing synchronization, and test result collection; alternatively, the computer device 10 can be used to host and manage a virtual environment, which includes multiple nodes to be tested.

[0030] Taking the communication testing method provided in this application embodiment as an example, the computer device 10 can construct a test environment for multiple nodes to be tested. Constructing the test environment includes obtaining and parsing test configuration information such as configuration parameters and resource information corresponding to each node to be tested, and performing test configuration operations such as logging into each node to be tested. Optionally, the computer device 10 issues container instance creation instructions to multiple nodes to be tested. After creating the container instance corresponding to the test environment on each node to be tested, each node to be tested performs pre-tests and multi-node communication tests through the container instance. Pre-tests are performed on multiple nodes to be tested in the test environment. The pre-tests are used to check the communication environment of each node to be tested before performing multi-node communication tests corresponding to multiple nodes, and to identify single-node communication problems that may affect multi-node communication tests. If the pre-tests pass, multi-node communication tests are performed on multiple nodes to be tested to obtain performance data.

[0031] Schematic, computer device 10 and multiple nodes under test jointly construct a test environment; in the test environment, pre-tests are performed on each of the multiple nodes under test to check the network connectivity and resource availability of the nodes under test; if the pre-tests pass, multi-node communication tests are performed on the multiple nodes under test to obtain performance data, which is used to characterize the communication performance between the multiple nodes under test. Based on this performance data, performance analysis, cluster design updates, and task allocation decisions based on network planning can be performed on the heterogeneous machine learning communication platform cluster corresponding to the multiple nodes under test.

[0032] Optionally, the computer device 10 may include any computer device such as a terminal or server, and this embodiment of the application does not limit this.

[0033] The aforementioned terminal is optional and can be a desktop computer, laptop computer, mobile phone, tablet computer, e-book reader, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, smart TV, smart vehicle, and other types of terminal devices. This application embodiment does not limit the specific terminal device to these types.

[0034] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0035] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0036] In some embodiments, the server described above can also be implemented as a node in a blockchain system.

[0037] For illustrative purposes, please refer to the following: Figure 2 This document illustrates a flowchart of a communication testing method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both simultaneously. This embodiment uses the execution of the method by a server as an example for illustration. Figure 2 As shown, the method includes at least one of the following steps.

[0038] Step 210: Build the test environment.

[0039] The test environment includes multiple nodes to be tested. In some embodiments, the test environment is constructed based on test requests created by testers for the heterogeneous machine learning platform cluster to be tested. Optionally, the test requests may also be requests generated by the heterogeneous machine learning platform cluster corresponding to the multiple nodes to be tested before executing machine learning tasks; this embodiment of the application does not limit this.

[0040] The testing environment includes computer hardware, software environment, testing tools, and nodes under test. The computer hardware and software environment includes, but is not limited to, the hardware and software environment corresponding to the node under test, and the hardware and software resources corresponding to the testing system used to control the execution flow of the testing tasks. It is worth noting that the relationship between the node under test and the testing system is not limited in this application embodiment. Optionally, the testing system can be implemented on the node under test in the testing environment, or it can be implemented as a terminal or server in the testing environment that has connections to multiple nodes under test.

[0041] A node to be tested refers to the entity being tested in the test environment, i.e., the target object of the test operation. Optionally, the node to be tested is implemented as a heterogeneous machine learning platform. The heterogeneous machine learning platform is used to execute machine learning tasks through communication between at least two computing units, and a cluster of heterogeneous machine learning platforms corresponding to multiple nodes to be tested is used to perform distributed computing. The process of performing distributed computing based on the heterogeneous machine learning platform cluster includes: splitting a computing task that meets at least one of the characteristics of large data scale, high computational complexity, high task parallelism, and large resource requirements into multiple subtasks; allocating the subtasks to the aforementioned heterogeneous machine learning platforms, and executing them collaboratively on the corresponding heterogeneous machine learning platforms through parallel execution or by arranging the execution flow of the subtasks according to the logical dependencies between tasks; obtaining the computation results corresponding to the multiple subtasks, summarizing and processing them to complete the overall computing task.

[0042] Alternatively, a computing unit can also be called an accelerator card, which refers to a hardware board designed to efficiently perform specific types of computing tasks, i.e., a processor. These specific types of computing tasks include, but are not limited to, graphics rendering tasks, scientific computing tasks, artificial intelligence training tasks, and inference tasks.

[0043] By building a test environment, the communication testing process can be adapted to different types of high-performance computing networks or communication interfaces, thereby improving the versatility of communication testing.

[0044] In some embodiments, the node under test includes at least two hardware devices for performing computing tasks, including but not limited to a central processing unit (CPU), GPU, GPGPU, ASIC, neural processing unit (NPU), tensor processing unit (TPU), vision processing unit (VPU), field-programmable gate array (FPGA), etc.

[0045] In some embodiments, a communication testing process based on a test environment is used to record or measure the communication behavior of the nodes under test, thereby evaluating the communication capabilities of the nodes under test. Accordingly, the test environment provides or reproduces the runtime environment corresponding to the execution of computational tasks for multiple nodes under test, serving as the basis for multi-node communication testing. The runtime environment corresponding to the nodes under test includes, but is not limited to, at least one of the following resources.

[0046] 1. Hardware Resources: Storage resources and network interfaces corresponding to the computing unit. Among them, storage resources include send and receive buffers required for allocating communication libraries, storage devices for storing software resources and test tools, etc. The implementation methods of storage resources include, but are not limited to, the memory space of the computing unit and the solid-state drive (SSD) corresponding to the node under test.

[0047] 2. Software Resources: These include, but are not limited to, an operating system, which provides low-level process management, memory management, and network protocol stack support during the testing process; drivers, which provide the operating system with interfaces to control and manage hardware devices; communication libraries, which encapsulate network communication-related programs; and testing tools, which provide communication testing programs. For example, the communication library includes Message Passing Interface (MPI), Unified Communication X (UCX), etc.

[0048] In some embodiments, software resources include runtime dependencies, external resources necessary for the operation of the operating system, drivers, etc. during the testing process (such as daemons that run independently of the communication behavior of the node under test), and basic software (such as drivers, programs for providing environment isolation and resource management, etc.).

[0049] 3. Resource Configuration: This section describes the parameters and settings of the runtime environment of the node under test. Resource configuration determines the resource allocation method and interaction behavior of the runtime environment through environment variables, network configuration, and permission configuration. Network configuration includes, but is not limited to, Internet Protocol (IP) addresses, hostname resolution for the node, and data transmission routing tables. Permission configuration includes, but is not limited to, memory access permissions, node login permissions, port access control, and other network resource permissions.

[0050] It is worth noting that the above description of the operating environment is only an illustrative example, and the embodiments of this application do not limit the specific implementation form of the operating environment of the node to be tested when constructing the test environment.

[0051] In some embodiments, the methods for building a test environment include, but are not limited to: directly installing the corresponding runtime dependencies on a physical device; or building a test environment in an isolated environment based on test configuration information, such as creating a corresponding virtual machine template based on a virtual machine control platform, obtaining an image container to create a corresponding environment instance; or automatically creating test scenarios by calling the corresponding application programming interface (API) of the cloud service programmatically.

[0052] The test configuration information describes the test environment; that is, it is a complete description of the runtime environment corresponding to the node under test. Based on the test configuration information, a test environment that meets the requirements of the communication test process can be established through at least one of the above-mentioned environment construction methods. The test configuration information includes, but is not limited to, the number and type of nodes under test, the configuration resources in the runtime environment of each node under test, the communication relationships between the nodes under test, and the test order and test strategies corresponding to pre-test and multi-node communication tests, respectively.

[0053] Indicatively, the communication relationships between the nodes to be tested include, but are not limited to, at least one of the following:

[0054] 1. Communication Mode: This indicates the organization and interaction method of data exchange between multiple nodes under test. It describes the sender and receiver information, node pair performance parameter types, and communication timing corresponding to the communication behavior. Communication modes between nodes under test include, but are not limited to, point-to-point communication, one-to-many broadcast communication, many-to-many reduction communication, and many-to-many full-switching communication. Optionally, communication modes also include synchronous communication, where both parties need to perform handshake and waiting confirmation operations before communication; and asynchronous communication, where the sender can send node pair performance parameters to the receiver without waiting for confirmation.

[0055] 2. Communication Protocol: This refers to the technical specifications, rules, and timing conventions that should be followed when multiple nodes under test exchange data. Illustratively, the content of the communication protocol includes, but is not limited to, the data format for performance parameters, the mechanisms for establishing and releasing connections during communication, and whether the communication is reliable.

[0056] The communication protocols between the nodes under test include, but are not limited to, InfiniBand (IB) technology, Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Remote Direct Memory Access (RDMA).

[0057] 3. Communication topology: Used to indicate the logical connection structure between multiple nodes under test, including but not limited to tree structure, mesh structure, ring structure, and multi-level structure that combines high-speed interconnection within the nodes under test with network interconnection between the nodes under test.

[0058] 4. Communication Dependency: This indicates the constraints on the execution order of multiple nodes under test during communication. A constraint means that the communication behavior of a node under test must wait for other specific nodes to complete their communication behavior before it can proceed. The reasons for such constraints include, but are not limited to: a node under test needs to wait for output data sent by at least one specific node before it can perform a receiving operation. For example, node B needs to receive the gradient value calculated by node A before it can perform a parameter update. That is, node B will block and wait to perform the receiving operation of gradient value data if node A has not sent gradient value data.

[0059] It is worth noting that the above classification of communication relationships is only an illustrative example. The embodiments of this application do not limit the specific implementation of the communication relationships between multiple nodes to be tested, nor the test configuration information used to build the test environment.

[0060] Step 220: Perform pre-tests on multiple nodes to be tested in the test environment.

[0061] Pre-testing is used to check the network connectivity and resource availability of the node to be tested.

[0062] Among them, the network connectivity within the test node refers to whether the data path between at least two computing units in the test node, and between the computing unit and the memory system of the test node, is reachable, and whether the data transmission capability meets the expectations of multi-node communication testing; illustratively, by running the inter-GPU link test program, it is possible to detect whether the link status between GPU hardware in the test node remains normal, such as obtaining a detection result of "active" indicating that the link between GPUs is available.

[0063] Network connectivity between nodes under test refers to whether the data path between any two nodes under test remains interconnected and whether the data transmission capability meets the expectations of multi-node communication testing. The data path includes the data path corresponding to the basic network and the data path corresponding to the high-performance computing network.

[0064] To illustrate, the basic network is used to carry data between nodes under test that is less sensitive to latency and relies on the software protocol stack in the communication library. Data transmitted through the basic network usually needs to be processed by the operating system kernel, such as API call requests with small data packets and high reliability requirements, as well as non-real-time, periodically collected log data. On the other hand, the high-performance computing network is used to carry data that is more sensitive to latency and has high throughput requirements. Data transmitted through the high-performance computing network usually needs to bypass the operating system kernel to reduce the number of data copies, reduce packet loss rate, and improve network performance, such as data transmission efficiency, such as gradient synchronization, model parameter and other computational data with high transmission frequency and low latency, as well as high-throughput, continuous real-time communication stream data.

[0065] Resource availability of a node under test refers to the state in which the hardware and software resources corresponding to the node under test are in a condition that can be normally requested, accessed, and used by the testing tools for multi-node communication testing. Factors affecting resource availability include, but are not limited to, the existence of resources, such as whether the network interface card device in the node under test can be identified; the sufficiency of resource capacity, such as whether the free memory in the node under test meets the buffer size required for multi-node communication testing; and the legality of access permissions, such as whether there are errors in the login authentication information of the node under test that prevent data from being read or written.

[0066] The resource availability of the node under test includes, but is not limited to, the availability of hardware links within the node, such as whether the network card is faulty, whether the link status is abnormal, etc., as well as computing resources (processor resources that generate node pair performance parameters), memory resources (used to cache node pair performance parameters), software resources (including but not limited to communication library resources), and permission resources. Hardware links refer to inter-card interconnect hardware links, such as Peripheral Component Interconnect Express (PCIe) links, which enable direct memory access between different hardware processors, reducing data transmission latency. Abnormal link status includes, but is not limited to, low transmission rate, high packet loss rate, and high transmission latency due to congestion.

[0067] In some embodiments, pre-testing is a multi-stage, automated testing operation targeting the nodes under test. It checks the test environment for communication problems that might cause multi-node communication tests to fail or malfunction, and which are not the focus of multi-node communication testing (network performance between multiple nodes under test). By performing pre-testing on the runtime environment of each node under test, the communication link status, driver status, and communication library integrity within the node under test are verified, preventing single-node communication problems from affecting the results of multi-node communication tests. Furthermore, by pre-testing for communication problems that may arise at different communication levels, the corresponding communication capability evaluation processes for different communication levels are differentiated. For example, the basic network or high-performance computing network between any two nodes under test can be tested separately, and the test results are recorded. This improves the localization of test anomalies and enhances the stability and accuracy of communication tests.

[0068] Schematic illustration: Pre-testing includes, but is not limited to, network connectivity testing, resource availability testing, and node-to-node communication testing between any two nodes under test. It is worth noting that this application does not limit the division between multiple stages of pre-testing; that is, for network connectivity and resource availability of the nodes under test, pre-testing at the same stage can check different categories or levels of communication problems. Schematic illustration: During network connectivity-related testing, the reachability of the basic network and high-performance computing network between any two nodes under test can be checked sequentially or simultaneously; or, during basic network connectivity testing, the availability of shared storage resources between multiple nodes under test can be checked.

[0069] In some embodiments, steps 210 and 220 are performed simultaneously. That is, during the automatic construction of the test environment, the network connectivity and resource availability of multiple nodes to be tested in the test environment are checked accordingly. Illustratively, the test environment is constructed such that, assuming basic network connectivity between any two nodes to be tested is successful, shared storage resources are defined at preset locations among the multiple nodes to be tested, such as mounting a shared directory. Read and write verification is then performed on the defined shared storage resources to test their availability.

[0070] Step 230: If the pre-test passes, perform multi-node communication tests on multiple nodes to be tested to obtain performance data.

[0071] Performance data is used to characterize the communication performance between multiple nodes under test, including but not limited to the data transmission efficiency between multiple nodes under test.

[0072] In some embodiments, performance data are metrics characterizing the communication capabilities of multiple nodes under test, including but not limited to: 1. Bandwidth data: The upper limit of the amount of data transmitted per unit time in the communication link between multiple nodes under test, usually measured in gigabit per second (Gb / s) or gigabyte per second (GB / s). In large-scale data transmission scenarios, such as gradient synchronization and model distribution, it determines the theoretical upper limit of data transmission speed; 2. Latency: The time interval from when a node sends a performance parameter to when the receiving node completes receiving it, usually measured in microseconds (μs) or milliseconds (ms). In scenarios with frequent interaction of small data packets, such as updating gradient values, it affects task execution efficiency; 3. Communication jitter: The standard deviation of communication latency or bandwidth data fluctuations over time, used to measure the stability of communication performance between multiple nodes under test; 4. Throughput: The total amount of data successfully transmitted per unit time, usually measured in GB / s or Gb / s. Bandwidth data is the theoretical upper limit of throughput, which reflects the real-time data transmission capability under test load; 5. Packet loss rate or retransmission rate: The proportion of data packets lost or corrupted during transmission out of the total number of data packets sent, or the proportion of data packets that need to be retransmitted due to packet loss out of the total number of data packets sent, used to indicate the quality of the communication network; 6. Time consumption: The time interval from the start of the communication behavior to the completion of the communication behavior.

[0073] Multi-node communication testing generates test loads and collects performance data generated during communication between multiple nodes under test under those loads. The multi-node communication test is performed using testing tools provided by a communication library. These tools automate the execution of multi-node communication tests, generate test results, and output them. Optionally, performance data can be obtained based on the test results, and the results can be analyzed to obtain preliminary structured test results. In some embodiments, the testing tools collect corresponding performance data during data exchange by executing different preset communication modes under different test loads.

[0074] In some embodiments, after the testing tool is configured on each node under test, the process of performing multi-node communication tests on multiple nodes under test using the testing tool includes, but is not limited to: creating and starting test processes simultaneously on multiple nodes under test using a parallel launcher based on executable files with consistent paths; obtaining execution parameters for the multi-node communication test, which are preset by the tester; execution parameters include, but are not limited to, the number of processes on each node under test, the number of communication groups for performing the multi-node communication test, the size of the communication groups and the communication mode corresponding to each communication group, and the list of nodes for parameter testing; optionally, the node list includes multiple nodes under test, or, if some nodes under test experience communication problems during the pre-test, the nodes with communication problems are excluded, and the node list includes at least two valid nodes, wherein the valid nodes are the nodes that have passed the pre-test among the multiple nodes under test.

[0075] The processes of the node under test are used to execute communication test tasks, including but not limited to initializing the communication library, establishing at least one communication group based on execution parameters, performing communication operations, and recording and collecting performance data during the multi-node communication test. Optionally, the number of processes of the node under test is consistent with the number of computing units included in the node under test, so that each computing unit can independently participate in the multi-node communication test.

[0076] In some embodiments, based on the execution parameters of the multi-node communication test, communication test operations are performed. Several communication operations are executed before the formal multi-node communication test to ensure that the hardware links of the multiple nodes under test are in a stable state. The formal multi-node communication test iteratively executes the communication behaviors corresponding to the communication primitives for different test loads, i.e., different data volumes. The communication primitives are indicated by the test tool and are used to instruct the most basic and indivisible communication operation units in the data exchange and collaborative processing operations between multiple nodes under test. During the formal multi-node communication test, performance data such as time consumption, bandwidth, and latency corresponding to each communication behavior are collected and output. Optionally, the test results corresponding to the multi-node communication test executed by the communication tool include the above performance data, the completion status of each communication operation (including success or failure status), the nodes under test where communication problems occur, and abnormal or error information.

[0077] The communication primitives include, but are not limited to: Send (a single node transmits node pair performance parameters to a specified target node), Receive (a single node retrieves node pair performance parameters from a specified source node), Broadcast (a single node distributes the same node pair performance parameters to multiple nodes), Gather (multiple nodes aggregate node pair performance parameters to a single node), Scatter (a single node distributes different node pair performance parameters to multiple nodes), AllGather (all nodes exchange with each other, and each node eventually has all node pair performance parameters), AllScatter (the inverse operation of AllGather), Reduce (multiple nodes perform node pair performance parameters according to specified operations, such as summation / maximization, and aggregate to a single node), AllReduce (the reduction result is broadcast to all nodes), ReduceScatter (the reduction result is sharded and distributed to each node), AllToAll (each node sends different node pair performance parameters to all other nodes), and Barrier (synchronous communication, i.e., execution continues only after all nodes have received or sent node pair performance parameters).

[0078] Optionally, the testing tools on each node to be tested can be deployed and configured during the test environment construction process, or during the pre-test to check the resource availability of the node to be tested, or before performing multi-node communication tests on multiple nodes to be tested.

[0079] If the pre-test passes, the abnormal or error information in the test results corresponding to the multi-node communication test excludes the situation where a single node under test has a communication problem or where there is a communication problem between any two nodes under test, thus limiting the abnormal or error information to the multi-node communication process.

[0080] In some embodiments, the performance data obtained by performing multi-node communication tests is a direct quantitative result of the communication capabilities of the heterogeneous machine learning platform cluster corresponding to multiple test nodes. Based on the performance data, it is possible to determine whether there are performance bottlenecks affecting the execution of machine learning tasks among the multiple test nodes, and to verify whether the communication path and topology meet the design expectations. Thus, based on the performance data, configuration parameters such as environment variables of the communication library can be adjusted, and computing resources and task processing scheduling can be planned for different computing units to improve the communication capabilities of the heterogeneous machine learning platform cluster.

[0081] In summary, the method provided in this application constructs a unified test environment to perform configuration processing on multiple nodes to be tested. Pre-tests and multi-node communication tests are automatically executed sequentially within this test environment. The pre-test serves as a preliminary check for multi-node communication testing. Once the pre-test passes (i.e., the test node's network is connected and resources are available), communication testing is performed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing repetitive debugging costs, manpower costs, and configuration time, thereby improving communication testing efficiency. Furthermore, it decouples communication problems on a single test node indicated by the pre-test from communication problems between at least two test nodes indicated by the multi-node communication test, enabling the location of communication anomalies to a single test node and improving the accuracy of communication testing.

[0082] In some embodiments, the above communication testing method is executed automatically based on test configuration information pre-set by the tester. Optionally, a test environment is constructed based on the test configuration information, and / or, pre-tests are performed on multiple nodes to be tested respectively based on the test configuration information, and / or, multi-node communication tests are performed on multiple nodes to be tested based on the test configuration information. Please refer to... Figure 3 , Figure 3 This is a flowchart of a communication testing method provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by both a terminal and a server simultaneously. This embodiment of the application illustrates the method as being executed by a server. Figure 3 As shown, step 210 above includes, but is not limited to, steps 212 to 214 below.

[0083] Step 212: Obtain test configuration information.

[0084] Test configuration information describes the test environment. This information is derived from a standardized configuration file pre-set by the tester. Upon receiving the standardized configuration file, the tester parses it to obtain the configuration information within.

[0085] Optionally, when receiving a test request corresponding to a communication test process, a unified configuration file pre-set by the tester is received as input data. The unified configuration file is a structured, fixed-format file, wherein the fixed format of the unified configuration file is set based on a summary of the communication test process over a historical period. For example, the file format includes, but is not limited to, YAML (Yet Another Markup Language Ain't Markup Language), JavaScript Object Notation (JSON), Extensible Markup Language (XML), etc., and this application embodiment does not limit this.

[0086] In some embodiments, the test configuration information includes, but is not limited to, at least one of the following.

[0087] 1. Network address information corresponding to the node under test: This information is used to uniquely identify the location of the node in a network composed of multiple nodes under test, thereby achieving hierarchical identification for data routing and end-to-end communication. Illustratively, the network address information corresponding to the node under test includes, but is not limited to, the physical layer Media Access Control Address (MAC), the network layer IP address, the transport layer port number, and the application layer hostname or domain name. When the network address information undergoes format validation, it is also used to generate a node list.

[0088] 2. Login authentication information for the nodes to be tested: The login method and authentication information for the nodes to be tested. The login method includes, but is not limited to, configuring a Secure Shell (SSH) key pair, configuring passwordless login based on the SSH key pair, and achieving automatic node login during the test by specifying a command. For example, the command executed is "ssh user@ip_address"; or, performing parallel login operations on multiple nodes to be tested based on SSH using batch operation and maintenance tools; or, performing login operations through the terminal window provided by the World Wide Web (Web) interface of the cloud platform or cluster management software.

[0089] The authentication information includes, but is not limited to, static usernames and passwords; SSH key pairs for passwordless login using public and private keys, or the private key information therein; two-factor authentication information combining static passwords and dynamic tokens; and authentication information corresponding to the bastion host, which first logs into the preset bastion host and then forwards the login to the node to be tested through the bastion host.

[0090] 3. Resource descriptions for nodes under test: Based on the resource descriptions, obtain the hardware and software resources or resource descriptions in the heterogeneous machine learning platform corresponding to each node under test, and build the test environment.

[0091] 4. Path information for the shared storage environment: The shared storage environment is used to store test configuration information, pre-test results, and performance data. The path information for the shared storage environment includes the mount point of the shared directory.

[0092] Optionally, the implementation methods of the shared storage environment include, but are not limited to: 4.1. Parallel file system, which supports multiple test nodes to concurrently read and write the same file through multiple paths and ensure data consistency; 4.2. Object storage interface, which stores and accesses data based on data, metadata, and globally unique identifiers; 4.3. Distributed storage engine, which integrates the physical storage resources of multiple test nodes into a unified storage pool to manage data distribution and replication; 4.4. Network storage protocol, which defines the communication rules and standards for transmitting data to be stored in the network, thereby realizing a network file system that allows test nodes to access files and directories on another test node through the network, such as remotely mounted directories (Network File System, NFS), shared folders (Server Message Block, SMB), network volumes (Apple Filing Protocol, AFP), etc., where the shared directory is the access point of the network file system presented by the network storage protocol on the test node, that is, the network storage protocol is the communication mechanism for realizing directory sharing and data access.

[0093] 5. Container Image Information: Indicates the image name corresponding to the image container used to provide the test environment. The corresponding image contains a file system snapshot of the complete software stack required to perform communication tasks, including but not limited to the basic operating system environment, the executable file of the multi-node communication test program corresponding to the test tools, the library files that the multi-node communication test program depends on, the fixed configuration files corresponding to the multi-node communication test, and other test tools.

[0094] 6. Test scale parameters and test strategy parameters for multi-node communication testing: Test scale parameters indicate the resource usage of multi-node communication testing, such as the data volume corresponding to the test load, the number of nodes to be tested, the number of requests to execute multi-node communication tests in batches, the number of communication groups and the size of the communication groups, etc. Test strategy parameters indicate the test behavior and test mode, such as the communication mode corresponding to different communication groups, the proportion of different communication modes, the test duration, etc.

[0095] Step 214: Based on the test configuration information, build the test environment.

[0096] In some embodiments, the communication testing process is executed automatically based on test configuration information. That is, in addition to the test environment construction step, the pre-test step and the multi-node communication test step also require the use of test configuration information. Optionally, the methods for using test configuration information in the pre-test step and the multi-node communication test step include, but are not limited to, obtaining configuration information such as parameter information and path information that are implicitly contained in the test environment; or, reading the explicit configuration information corresponding to the test configuration information; or, receiving test instructions generated by the control module based on the test configuration information. The control module is a module in the test system used for task scheduling and resource management. The control module can be part of the test environment, parsing the unified configuration file to obtain the test configuration information during the test environment construction process, and generating test instructions corresponding to the test configuration information in real time throughout the entire communication testing process.

[0097] Optionally, building the test environment based on the test configuration information includes installing runtime dependencies. These runtime dependencies include, but are not limited to, software packages, library files, and specific versions of system components that the multi-node communication test program must rely on during the execution of multi-node communication tests. Illustratively, installing runtime dependencies includes automatic installation via the operating system's package manager and the programming language's package management tools: first, identifying runtime dependencies; based on the test configuration information, obtaining the runtime dependencies and their corresponding version information; configuring the software source repository to ensure that runtime dependency packages are obtained; installing using package manager commands; and checking whether the runtime dependencies are successfully installed and whether the versions are correct.

[0098] Once the test environment is built, a unified entry point, command, or scheduling system automates the process of starting communication test procedures and other necessary components across multiple test nodes. Optionally, the communication test procedure includes performing pre-test steps on each test node and multi-node communication test steps on all test nodes. This avoids the tedious manual login to each test node and the separate initiation of the communication test procedure, ensuring that the communication test procedures of multiple test nodes start collaboratively in the correct order and with the correct parameters, thus bringing the entire distributed test task into operation. The start order and parameters are obtained based on the test configuration information.

[0099] Based on the image information in the test configuration, container instances corresponding to the test environment are created for each of the multiple nodes to be tested. The container instances are used to provide the execution environment for pre-testing and multi-node communication testing. Optionally, the image information includes, but is not limited to, the image name and the image file storage path.

[0100] A container instance is an isolated, independent runtime environment. A container image is a static, read-only collection of files containing all the information needed to start and run a specific operating system, such as file system structure, system configuration, multi-node communication test programs, runtime dependencies, and library files. A container image is a template for creating a running instance, or container. Compared to virtual machine images, container images are lightweight, executable software packages used to ensure the consistency of multi-node communication test programs across different environments. Optionally, a container instance can be implemented as a Docker container.

[0101] Optionally, when the container instance corresponds to the runtime environment of the multi-node communication test program, the pre-test can be performed outside the test environment provided by the container instance; the container image can also be built based on the runtime environment required by the test program corresponding to the pre-test, and the pre-test and the test environment required by the multi-node communication test do not conflict. This application embodiment uses the example of a container image built based on the runtime environment corresponding to the multi-node communication test program.

[0102] In some embodiments, the process of creating a container image includes, but is not limited to, obtaining a container image containing the underlying operating system as the base image; installing specific versions of library files, compilers, and runtime dependencies corresponding to the multi-node communication test program into the base image using container build tools; installing hardware support libraries corresponding to the computing units in the node under test (the container image does not include hardware drivers, which are accessed through the host kernel and drivers of the node under test after instance); copying the executable file or code corresponding to the multi-node communication test program into the base image and configuring the default execution command; and publishing the built container image to a container repository, which provides the image corresponding to the image information in the test configuration information to the node under test. Optionally, the software environment library corresponding to the multi-node communication test program can be obtained based on testing experience within a historical time period, or based on fixed settings corresponding to the communication library, or customized by the tester based on the multi-node communication test to be executed.

[0103] Multi-node communication testing involves multiple nodes working collaboratively. Each node needs to run a multi-node communication test program. Container instances, serving as the test environment for this program, must be started on each node. Each node pulls the corresponding image from the container repository and runs the container through instantiation. By running containers created from the same image on each node, the file system layer of each container instance contains the same software dependency repository, ensuring complete consistency in software dependencies, library versions, and system configurations across all nodes, thus eliminating the interference of environmental differences on test results. The control module or test system uniformly issues commands to each node to start or manage the container instances on their respective nodes.

[0104] This illustration demonstrates how container instances corresponding to multiple test nodes interact across nodes. Each container instance encapsulates the exact same version of communication libraries (such as MPI), runtime dependencies, etc., eliminating communication compatibility issues caused by differences in library versions and other software environments between test nodes. Container instances connect to the host network of their respective test nodes via virtual network interfaces (such as host network mode), allowing communication between container instances on multiple test nodes to function like communication between physical machines, while maintaining network configuration consistency, such as unified port mapping rules. Hardware resources obtained from each container instance's perspective, such as compute unit numbers, can be uniformly organized through test configuration information, ensuring that distributed computing tasks have the same resource access logic across all test nodes.

[0105] For illustrative purposes, please refer to the following: Figure 4 This illustrates a schematic diagram of the process for creating a container instance according to an exemplary embodiment of this application. Figure 4 As shown, image 420 is obtained by copying from source file 410, where the source file is the software runtime environment corresponding to the multi-node communication test program. Based on image 420, instantiation operations are performed on multiple nodes to be tested to create at least one container instance. For example, the first container instance 431 provides a test environment for the first node to be tested, and the second container instance 432 provides a test environment for the second node to be tested.

[0106] By automatically and uniformly starting container instances on multiple test nodes to execute communication test processes, the consistency of the software environment among multiple test nodes is ensured, avoiding the problem of unstable test results caused by differences in the test environment between different test nodes, eliminating the uncertainty caused by environmental differences, and using a unified container image to ensure that the test environment on each node is completely consistent during testing on multiple test nodes, thereby improving the stability of communication tests.

[0107] In some embodiments, when test configuration information is obtained, a shared storage environment is created among multiple nodes to be tested based on the path information in the test configuration information. The shared storage environment is used to store the test configuration information, the test results corresponding to the pre-test, and performance data. Illustratively, the creation of the shared storage environment includes creating and mounting a shared directory. Taking NFS as an example, a mount point is created at the mount location of the shared directory indicated by the test configuration information, such as `mkdir / mnt / shared`, where the mount point path is ` / mnt / shared`. A mount command is executed to mount the shared directory to the local mount point corresponding to the node to be tested, such as `mount server_ip: / shared_data / mnt / shared`, where `server_ip` is the IP address of the NFS server storing the physical directory corresponding to the shared directory, and `: / shared_data` is the path of the shared directory on the server side. Optionally, automatic mounting at boot can be configured by editing the system file table configuration file.

[0108] Before mounting shared files, a shared directory needs to be created on the server side, including but not limited to the following steps: Create the physical directory to be shared, such as `mkdir / shared_data`; Edit the NFS configuration file, such as ` / etc / exports`, to define which test nodes can access each part of the directory with what permissions; Add sharing rules for the shared directory, such as ` / shared_data client_ip(rw,sync)`, where `client_ip` is the IP address of the test node allowed to access the shared directory, `rw` indicates read and write permissions, and `sync` indicates synchronous writing, meaning that the corresponding test node will only respond after the data is confirmed to be written to disk; Apply the above configuration to make the sharing effective, such as `exportfs -a`, where `-a` is an option parameter used to export at least one shared directory defined in ` / etc / exports`.

[0109] Optionally, a shared directory can be created before building the test environment and before obtaining the test configuration information. This shared directory can be created in advance for multiple test nodes. Additionally, the server can be one of the test nodes, or a computer device connected to a heterogeneous machine learning platform cluster corresponding to multiple test nodes.

[0110] In some embodiments, creating a shared storage environment requires consideration of network connectivity between multiple nodes under test. Therefore, if pre-tests performed on each of the multiple nodes under test pass, the shared storage environment is created based on the test configuration information. Optionally, in the pre-test, after checking the network connectivity between any two nodes under test, the shared storage environment can be created if the network between any two nodes under test is reachable.

[0111] By creating a shared storage environment among multiple test nodes, it ensures that multiple test nodes access the same data, avoiding synchronization issues between multiple copies of the data. This enables unified data management during the testing process, eliminating the need to operate on each test node individually. It improves the configuration efficiency of communication testing, thereby increasing the efficiency and simplicity of test environment construction. It also reduces the complexity of performance parameters and test configuration information for distribution nodes in the communication testing process. Furthermore, it enables concurrent access from multiple test nodes, saves storage space, improves the scalability of test nodes, decouples computing resources from storage resources, facilitates the storage of test results and log files generated during the testing process, and facilitates unified collection of test results.

[0112] In summary, the method provided in this application constructs a unified test environment to perform configuration processing on multiple nodes to be tested. Pre-tests and multi-node communication tests are automatically executed sequentially within this test environment. The pre-test serves as a preliminary check for multi-node communication testing. Once the pre-test passes (i.e., the test node's network is connected and resources are available), communication testing is performed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing repetitive debugging costs, manpower costs, and configuration time, thereby improving communication testing efficiency. Furthermore, it decouples communication problems on a single test node indicated by the pre-test from communication problems between at least two test nodes indicated by the multi-node communication test, enabling the location of communication anomalies to a single test node and improving the accuracy of communication testing.

[0113] By using unified test configuration information to describe the information of the nodes under test in the test environment, the login authentication information of each node under test, and other configuration parameters, the test environment is constructed and the task flow for subsequent pre-testing and multi-node communication testing is organized. On the one hand, this avoids testers manually participating in the specific communication test process in multiple stages, improving the overall controllability and efficiency of communication testing. On the other hand, based on the test configuration information, the test environment and test strategy can be dynamically adjusted when the communication test process or requirements change, such as changing the scale of the nodes under test, changing the task execution order of the multi-stage communication test process, or changing the test configuration parameters. This allows for flexible adaptation to different heterogeneous machine learning platform clusters and the network topology corresponding to different communication strategies, exhibiting good scalability and universality, and improving the flexibility and scalability of communication testing methods.

[0114] In some embodiments, pre-testing of the nodes under test includes multi-stage test tasks, such as network connectivity testing performed on any two nodes under test, resource availability testing performed on a single node under test, and node-to-node communication testing performed on any two nodes under test. Please refer to [reference needed]. Figure 5 , Figure 5 This is a flowchart of a communication testing method provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by both a terminal and a server simultaneously. This embodiment of the application illustrates the method as being executed by a server. Figure 5 As shown, step 220 above includes, but is not limited to, steps 222 to 226 below.

[0115] Step 222: Perform a network connectivity test on any two of the multiple test nodes.

[0116] Network connectivity testing is used to check the network connectivity of the nodes under test. The network connectivity of the node under test includes both internal network connectivity and inter-node network connectivity. Internal network connectivity refers to whether the data path between at least two computing units within the node under test, and between the computing unit and the node's memory system, is reachable, and whether the data transmission capability meets the expectations of multi-node communication testing. Inter-node network connectivity refers to whether the data path between any two nodes under test remains interconnected and whether the data transmission capability meets the expectations of multi-node communication testing. The data path includes the data path corresponding to the underlying network.

[0117] In some embodiments, the business network between multiple nodes under test is the network infrastructure that carries the actual business application data traffic of users. It is used to support data exchange, service calls, and user access between applications involved in the execution of machine learning tasks by the heterogeneous machine learning platform cluster. Optionally, the business network includes the basic network. In scenarios where communication testing processes are performed on multiple nodes under test, the business network is used to transmit specific business data corresponding to the test, including but not limited to API requests, database query data, file transfers, microservice communication, etc.

[0118] Optionally, in the test environment, the business network is usually isolated from the management network, storage network, and high-performance computing network by physical or logical means to avoid interference between performance parameters caused by different nodes in the test process.

[0119] In some embodiments, network connectivity testing includes at least one of the following steps: checking whether the data paths corresponding to the underlying networks between the test system and each node under test, and between multiple nodes under test, are reachable; performing login verification on each node under test, optionally verifying whether the user's permissions can be normally executed for subsequent test tasks based on the login verification information in the test configuration information; if the above communication link check and login verification are passed, creating a shared storage environment at a specified location among multiple nodes under test, optionally creating a shared storage environment or mounting a shared directory based on the path information corresponding to the shared storage environment in the test configuration information; performing read / write verification on the shared storage environment, checking the availability of shared storage resources, and ensuring that the nodes under test can normally access the shared storage resources.

[0120] Optionally, in addition to obtaining the path information corresponding to the shared storage environment based on the test configuration information, it is also possible to obtain the configuration information corresponding to the shared storage environment, such as the information of at least two nodes under test that can access the shared storage environment, and the way the nodes under test access the shared storage environment.

[0121] To illustrate, login verification involves the test system attempting to log in to each node under test via SSH key before distributing test data to multiple nodes under test, based on test requirements. The authorization key file of each node under test receives the SSH key, so that the process corresponding to the test task can perform the test data distribution operation with the expected user identity, in order to verify whether the login function is correct and whether the user permissions are correct.

[0122] In some embodiments, the number of multiple nodes to be tested is N, where N is an integer greater than 1; each node to be tested sends test data to N-1 other nodes to be tested, and the test data is used to test the availability of the communication link between the node to be tested and other nodes.

[0123] Schematic illustration: Detecting network connectivity between multiple test nodes, i.e., detecting network connectivity between any two test nodes, includes, but is not limited to: obtaining a set of node pairs consisting of any two test nodes, where the set includes at least three node pairs, and each node pair includes two different test nodes. That is, performing network connectivity tests from each test node to N-1 other test nodes (excluding itself), where N is the number of test nodes and is an integer greater than 1. The detection steps include at least one of the following.

[0124] 1. Based on one of the nodes to be tested in any node pair in the node pair set, use a network diagnostic tool to send network protocol data packets to the other node to be tested in the same node pair to determine the network reachability between the node pairs. The network diagnostic tool includes the Packet Internet Groper (PING) command based on the Internet Control Message Protocol (ICMP). Optionally, the test data includes network protocol data packets.

[0125] 2. Based on one of the nodes to be tested in any node pair in the node pair set, use the corresponding dedicated testing tool for port detection to verify whether the specified port of the other node to be tested in the same node pair is open and can successfully establish a network connection.

[0126] 3. In a fully interconnected mode among multiple test nodes, sequentially perform the above operations on each test node in batches, checking that the network connection between any two test nodes is normal. Illustratively, the test results corresponding to the basic network connectivity test include abnormal network conditions, including but not limited to at least one of the following situations.

[0127] 3.1 Network unreachable: This includes situations such as no response to requests and timeouts. Possible causes include, but are not limited to, missing routes, firewall blocking, and system crashes of the node under test. 3.2 Packet loss: This includes situations such as partial unresponsive requests and intermittent packet loss. Possible causes include, but are not limited to, communication link congestion, hardware failure, and signal interference. 3.3 High latency: This includes situations such as response times far exceeding the expected baseline. Possible causes include, but are not limited to, route detours, low link quality, and processing delays of intermediate devices. 3.4 Out-of-order or duplicate transmission: This includes responses arriving out of order or repeatedly. Possible causes include, but are not limited to, multipath transmission and abnormal link-layer retransmission mechanisms. 3.5 Port unreachable: This includes situations such as ICMP port unreachable or connection requests being rejected. Possible causes include, but are not limited to, the corresponding communication service not being started or firewall ports not being open.

[0128] In some embodiments, if the network connectivity test fails, the communication test process is terminated, or the node to be tested that fails the pre-test is marked and excluded from the multi-node communication test scope. Optionally, after completing the network connectivity test of the node to be tested, the test results are stored in a shared network storage environment, wherein the test results include, but are not limited to, basic network communication performance data, whether the network status is normal, and the reasons for communication anomalies that can be collected.

[0129] By performing network connectivity tests on any two nodes among multiple test nodes, critical business services, such as API gateway services and database services, are automatically verified to be accessible at the network layer. If anomalies are detected in the communication link, the communication path is adjusted or the communication test process is terminated to prevent network interruptions that could render critical business services unavailable and affect the accuracy and stability of multi-node communication testing. The purpose of multi-node communication testing is to evaluate the communication capabilities between multiple test nodes. By using network connectivity testing to identify communication problems related to network connections before multi-node communication testing, it ensures that multi-node communication testing can be executed normally. From a holistic perspective, this reduces the accuracy of the evaluation and improves the distinguishability of fault causes and fault areas, preventing misdiagnosis of network problems as resource problems or vice versa, thus improving the efficiency of communication testing.

[0130] Step 224: Perform resource availability tests on multiple nodes to be tested.

[0131] Resource availability testing is used to check the resource availability of the node under test. This resource availability includes, but is not limited to, ensuring that the hardware and software resources corresponding to the node under test are in a state where they can be normally requested, accessed, and used by the testing tools for multi-node communication testing. Specifically, this includes the availability of hardware links within the node under test, such as whether the network card is faulty or whether the link status is abnormal, as well as computing resources (processor resources that generate node pair performance parameters), memory resources (used to cache node pair performance parameters), software resources (including but not limited to communication library resources), and permission resources. Optionally, the resource availability of the node under test also includes whether the communication environment within a single node is available, i.e., the network connectivity within the node under test.

[0132] In some embodiments, performing resource availability testing on multiple nodes under test includes one or more of the following: performing hardware verification processing on multiple nodes under test, wherein the hardware verification processing is used to check the hardware availability of the nodes under test; performing software verification processing on multiple nodes under test, wherein the software verification processing is used to check the driver availability of the nodes under test; and performing single-node communication testing on multiple nodes under test, wherein the single-node communication testing is used to check the compatibility between the hardware and driver of the nodes under test.

[0133] Optionally, single-node communication testing is performed using corresponding testing tools, such as a single-node communication test program. Illustratively, assuming the hardware of the node under test can be correctly identified by the test system, the device model meets expectations, the driver is loaded, the driver version is compatible with the hardware, and read / write permissions for the hardware device are available, communication behavior tests such as data copying are performed on at least two computing units within the same node under test to verify that the communication function between computing units, such as GPUs, is normal; basic tests in the communication library are run; loopback tests are performed on the local network card using the IB bandwidth test tool; for different test loads and corresponding data volumes, the measured values ​​of the transmission bandwidth between computing units in the node under test are measured, and the measured values ​​are compared with the theoretical values, where the theoretical value is the upper limit of the transmission bandwidth set through theoretical estimation. The above single-node communication test is performed on any one of multiple nodes under test to detect whether there are communication problems within the node under test.

[0134] In some embodiments, resource availability testing includes at least one of the following steps: diagnosing the hardware status of the heterogeneous machine learning platform corresponding to the node under test; optionally, based on the node configuration data corresponding to the node under test in the test configuration information, obtaining the link status and availability between computing units or network cards, and ensuring that the node under test can be identified by the test system; diagnosing the driver and communication library loading status corresponding to the node under test, and checking whether the software compilation environment and software stack loading required for multi-node communication testing are complete; obtaining a single-node communication test program, performing communication test operations within the scope of a single node under test, and verifying whether the operating environment of the node under test is correct; optionally, based on the network address in the test configuration information, automatically obtaining the single-node communication test program from the network, or, based on the test configuration information, obtaining and deploying the single-node communication test program when building the test environment, and compiling and running the single-node communication test program during the execution of resource availability testing.

[0135] Optionally, the above resource availability test includes network connectivity testing within the node under test. This application embodiment does not limit which pre-test stage the network connectivity testing within the node under test is in.

[0136] In some embodiments, if the resource availability test fails, the communication test process is terminated, or the node to be tested that fails the pre-test is marked and excluded from the multi-node communication test scope. Optionally, after completing the resource availability test of the node to be tested, the test results are stored in a shared network storage environment, wherein the test results include, but are not limited to, whether the internal hardware and software status of the node to be tested is normal, whether the hardware and driver are compatible, and whether the connectivity between hardware is as expected.

[0137] Before performing multi-node communication tests on multiple nodes, the system automatically checks the operating environment of each node under test, decoupling single-node communication issues from multi-node communication issues. This eliminates abnormal or faulty factors that could interfere with the overall communication test results at the single-node level, ensuring that the nodes under test have the basic conditions to participate in multi-node communication tests. This improves the efficiency and accuracy of communication tests and enhances the precision of communication problem localization.

[0138] Step 226: If the network connectivity test and resource availability test pass, perform a node-to-node communication test on any two nodes among the multiple nodes to be tested to obtain the node-to-node communication test results.

[0139] The node-to-node communication test results indicate whether the communication capability between any two nodes under test meets the requirements of multi-node communication testing. After passing network connectivity and resource availability tests, the test system establishes a communication link between any two nodes under test according to a preset test strategy and executes the node-to-node communication test.

[0140] In some embodiments, node-to-node communication test results are used to examine the communication capabilities between the high-performance computing networks upon which multi-node communication tests rely. Illustratively, the base network ensures the availability, security, and manageability of the network between nodes under test during concurrent access by multiple services. It typically carries data that is insensitive to latency, has moderate throughput requirements, and relies on software protocol stack processing. Node-to-node performance parameters transmitted by the base network typically pass through the TCP / IP protocol stack and are processed by the CPU, allowing millisecond-level latency. The high-performance computing network ensures low latency and high throughput when multiple nodes under test perform distributed computing tasks. It typically carries data that is extremely sensitive to latency, has extremely high throughput requirements, and needs to be accessed directly without bypassing the kernel. Node-to-node performance parameters transmitted by the high-performance computing network typically bypass the operating system kernel and are processed directly by the network card hardware, requiring microsecond-level latency and a network environment with no packet loss.

[0141] In some embodiments, node-to-node communication testing for any two nodes under test includes, but is not limited to, establishing a communication link between any two nodes under test; obtaining node-to-node performance parameters based on the communication link, whereby the node-to-node performance parameters characterize the network performance between the two nodes under test; and obtaining node-to-node communication test results based on the node-to-node performance parameters, whereby the node-to-node communication test results indicate whether the node-to-node performance parameters have reached a preset network performance threshold. The establishment of the communication link is based on a preset test strategy in the test system; optionally, the test strategy can be obtained by parsing test configuration information.

[0142] In some embodiments, during communication between two nodes under test, a TCP / IP network connection is first established through a basic network handshake, enabling the exchange of control information between the two nodes under test, confirming the existence of the other party, negotiating communication parameters, and exchanging address information required by the high-performance computing network, thus ensuring the reliability of control command transmission. Secondly, if the basic network handshake is successful, performance parameters are transmitted to the nodes of large-scale parallel computing through the high-performance computing network, achieving high-bandwidth and low-latency data exchange on a high-speed channel.

[0143] In some embodiments, before establishing the communication link corresponding to any two nodes under test in the high-performance computing network, a set of node pairs consisting of any two nodes under test is obtained from a plurality of nodes under test. The set of node pairs includes at least three sets of node pairs, and each node pair includes two different nodes under test. That is, node pair communication tests are performed based on each node under test to N-1 other nodes under test, where N is the number of multiple test nodes and N is an integer greater than 1.

[0144] To illustrate, the steps for establishing a communication link include automatically initiating the transmission of node pair performance parameters between the node pairs. Taking the two nodes to be tested, A and B, as an example, node A designates a high-performance network interface card (NIC) on the node to be tested as a server device. This server remains running and is ready to receive connection requests from other NICs. Node B designates a corresponding high-performance NIC as a client device. Through an automated testing tool, such as the IB driver test program, the client of node B initiates a connection request to the server of node A. If the server of node A receives and agrees to the connection request, node A sends a connection feedback to node B, thereby establishing a communication link.

[0145] During the node-to-node communication test, it is necessary to connect the high-performance network cards of nodes A and B to establish a one-time communication link. After the node-to-node communication test, the one-time communication connection between nodes A and B will be disconnected, that is, the network cards corresponding to the client and server will be turned off. In the next node-to-node communication test, the network cards will be replaced or the node to be tested will establish a new communication link.

[0146] In some embodiments, based on the aforementioned communication link, node-to-node communication tests are performed on the node pairs to obtain node pair performance parameters, such as bandwidth data, latency, and other indicators characterizing communication capabilities. This allows for the determination of whether the communication capabilities of the high-performance computing network meet the expected conditions for multi-node communication testing, and the acquisition of node-to-node communication test results. Optionally, the expected conditions for multi-node communication testing can be obtained by parsing test configuration information, or pre-configured in the test task execution logic of the test system.

[0147] In some embodiments, if a node-to-node communication test fails, the communication test process is terminated, or the node to be tested that fails the pre-test is marked and excluded from the multi-node communication test scope. Optionally, after completing the node-to-node communication test for any two nodes to be tested, the test results are stored in a shared network storage environment, wherein the test results include, but are not limited to, node-to-node performance parameters, and whether the node-to-node communication test results, i.e., the node-to-node performance parameters, meet the expected conditions of the multi-node communication test.

[0148] Before testing the communication bandwidth of multiple nodes under test, node-to-node communication tests are performed on any two high-performance computing (HPC) nodes to determine whether the HPC meets the expected conditions for multi-node communication testing. This checks the communication capabilities between multiple nodes under test, decoupling basic network communication issues from HPC communication issues. At the HPC level, abnormal or faulty factors that could interfere with the overall communication test results are eliminated, improving the success rate and stability of communication testing. This ensures that the nodes under test meet the communication capability requirements for participating in multi-node communication testing, improves the efficiency and accuracy of communication testing, and enhances the precision of communication problem localization.

[0149] Based on the test results corresponding to the pre-test, at least two valid nodes are obtained; these at least two valid nodes are nodes that have passed the pre-test among multiple test nodes; multi-node communication tests are performed on these at least two valid nodes to obtain performance data. When some test nodes exhibit anomalies at any test stage, the test system automatically marks the test nodes that fail at least one stage of the pre-test as abnormal nodes and isolates or skips them in subsequent multi-node communication tests by excluding them from the multi-node communication test scope; alternatively, test nodes that pass the pre-test are marked as valid nodes, and multi-node communication tests are performed only on valid nodes that pass any stage of the pre-test. This avoids the impact of test nodes with communication problems on the accuracy of multi-node communication tests without needing to rebuild the test environment and perform the pre-test, thus improving the efficiency and stability of communication tests.

[0150] For illustrative purposes, please refer to the following: Figure 6 This illustrates a schematic diagram of the overall communication testing process provided in an exemplary embodiment of this application. Figure 6As shown, the communication testing process includes at least six stages of communication testing, wherein each stage of communication testing must be executed after the previous stage of communication testing is completed or the test is passed. Optionally, the specific processing operations in each stage of communication testing are as described in the above embodiments of this application. The communication testing operations in each stage are only illustrative examples, and the specific processing operations are not strictly limited. Step 610 is executed to obtain test configuration information. A unified configuration file is received and parsed to obtain test configuration information. The test configuration information is used to describe the test environment. Based on the test configuration information, a test environment including multiple stages to be tested is constructed.

[0151] like Figure 6 As shown, step 620 is executed, performing a network connectivity test on any two of the multiple test nodes. The network connectivity test is used to check the network connectivity of the test nodes. Specifically, the network connectivity of the test node includes the network connectivity within the test node and the network connectivity between test nodes. The network connectivity within the test node refers to whether the data path between at least two computing units within the test node and between the computing units and the memory system of the test node is reachable, and whether the data transmission capability meets the expectations of multi-node communication testing.

[0152] like Figure 6 As shown, step 630 involves performing resource availability tests on multiple nodes under test. The resource availability test checks the resource availability of the nodes under test, including but not limited to ensuring that the hardware and software resources corresponding to the nodes are in a state where they can be normally requested, accessed, and used by the testing tools corresponding to the multi-node communication test. Optionally, there may be no execution order dependency between steps 620 and 630. For example, step 620, performing network connectivity tests on any two nodes under test, can be performed before or after step 630, performing resource availability tests on each node under test. Similarly, step 620, performing network connectivity tests on any two nodes under test, and step 630, performing resource availability tests on each node under test, can be performed synchronously.

[0153] like Figure 6As shown, step 640 involves performing a node-to-node communication test on any two nodes from the multiple test nodes, obtaining the test results. Specifically, the node-to-node communication test is performed only after the network connectivity test and resource availability test have both passed. The node-to-node communication test results indicate whether the communication capability between any two test nodes meets the requirements of the multi-node communication test. If the network connectivity test and resource availability test have passed, the test system establishes a communication link between any two test nodes from the multiple test nodes according to a preset test strategy, and then performs the node-to-node communication test. Figure 6 As shown, step 650 is executed to perform multi-node communication tests on multiple nodes under test, and performance data is obtained. This performance data is used to characterize the communication performance between the multiple nodes under test.

[0154] like Figure 6 As shown, step 660 is executed to summarize the test results and output structured test results. The test results corresponding to at least one stage of the pre-test, along with at least one performance data point, are summarized. Data processing is performed on the test results to obtain structured information. Based on this structured information, structured test results corresponding to the multi-node communication test are output. This phased communication test process ensures that subsequent test stages are executed under the premise that the preconditions are met, avoiding overall communication test failure or distorted test results due to local communication problems.

[0155] In summary, the method provided in this application constructs a unified test environment to perform configuration processing on multiple nodes to be tested. Pre-tests and multi-node communication tests are automatically executed sequentially within this test environment. The pre-test serves as a preliminary check for multi-node communication testing. Once the pre-test passes (i.e., the test node's network is connected and resources are available), communication testing is performed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing repetitive debugging costs, manpower costs, and configuration time, thereby improving communication testing efficiency. Furthermore, it decouples communication problems on a single test node indicated by the pre-test from communication problems between at least two test nodes indicated by the multi-node communication test, enabling the location of communication anomalies to a single test node and improving the accuracy of communication testing.

[0156] By conducting multi-stage pre-tests, such as network connectivity testing, resource availability testing, and node-to-node communication testing, test results for each stage are generated. This enables the testing system to more accurately pinpoint the cause of communication problems when anomalies occur during multi-node communication testing. It can distinguish whether the anomaly originates from at least one of the following: the basic network, the single-node operating environment, the high-performance computing network, or the execution process of the multi-node communication test. Alternatively, it can prevent nodes with communication problems from participating in multi-node communication tests, thereby improving the efficiency and stability of communication testing.

[0157] Please refer to Figure 7 , Figure 7 This is a flowchart of a communication testing method provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by both a terminal and a server simultaneously. This embodiment of the application illustrates the method as being executed by a server. Figure 7 As shown, after step 230 above, there are including but not limited to the following steps 710 to 720.

[0158] Step 710: Perform data processing on at least one of the test results and performance data corresponding to the pre-test to obtain structured information.

[0159] In some embodiments, a multi-node communication test program is obtained, which is used to instruct the execution of multi-node communication tests. The multi-node communication test program is used to perform multi-node communication tests on multiple nodes under test to obtain performance data. Accordingly, the structure and type of the performance data are pre-specified by the execution logic of the multi-node communication test program, and the execution logic for data processing can be set based on the execution logic of the multi-node communication test program. Alternatively, the execution logic for data processing can also be set according to the performance data corresponding to multi-node communication tests within a historical time period and the format and structure of the test results corresponding to the pre-test.

[0160] Each testing phase includes at least one phase of the pre-test and a multi-node communication test. Accordingly, the test results of the communication test process include at least one test result corresponding to the pre-test and performance data corresponding to the multi-node communication test. For illustration, if the pre-test fails, the communication test process is terminated, and the test results of the communication test process only include the test results corresponding to the pre-test, excluding performance data. If at least one phase of the pre-test fails, the test results of the communication test process only include the test results corresponding to the pre-tests that have been executed, excluding the test results corresponding to the pre-tests that have not been executed. For example, if the resource availability test fails, the test results of the communication test process include the test results of the resource availability test and the test results of the network connectivity test executed before it, excluding the test results of the unexecuted node-to-node communication test.

[0161] Indicatively, the test results of the pre-test include, but are not limited to, basic network communication performance data, whether the network status is normal, the reasons for communication anomalies that can be collected, whether the internal hardware and software status of the node under test is normal, whether the hardware and driver are compatible, whether the connectivity between hardware meets expectations, the node pair performance parameters used to characterize network performance between any two nodes under test, and whether the node pair communication test results, i.e., the node pair performance parameters, meet the expected conditions of multi-node communication test.

[0162] In some embodiments, the method of obtaining at least one of the test results and performance data corresponding to the pre-test includes, but is not limited to, obtaining at least one of the test results and performance data corresponding to the pre-test from a shared storage environment among multiple nodes to be tested; or, obtaining the log file corresponding to the communication test process, and obtaining at least one of the test results and performance data corresponding to the pre-test based on the log file, wherein the location of the log file is specified by the test configuration information, and optionally, the log file may be stored in a shared storage environment.

[0163] In some embodiments, log files are used to indicate test operation records performed by the system under test during the communication test process. Indicatively, each log entry may include the operation type, related shared directory information, operation fields and values, operation time, associated test phase identifier, etc., but this application embodiment does not limit this. The test system traverses the log files on multiple nodes under test based on pre-set test logic and parses the corresponding structured information content based on the log file name and log content. Optionally, logs associated with the current test phase can be filtered from the log files. For example, the corresponding operation logs can be obtained by matching the test phase identifier, session identifier, or timestamp to obtain the operation records generated during the execution of that test phase.

[0164] Optionally, the logical process of obtaining structured information by performing data processing on at least one of the test results and performance data corresponding to the pre-test through the test system is based on the test results and performance data of the pre-test within a historical time period. The data aggregation and processing logic is automatically summarized by testers or machines and integrated into the corresponding module of the test system.

[0165] In some embodiments, the entity that performs data processing on at least one of the test results and performance data corresponding to the pre-test to obtain structured information is the test system. The test system is the entity that executes test control operations, and can be a local host, a management node among multiple nodes to be tested, or a continuous integration (CI) server. The test system is responsible for code compilation, test task scheduling, test result collection, and analysis. Correspondingly, the nodes to be tested are the entities that run the actual test tasks. Optionally, the nodes to be tested host test programs through container instances to execute specific test loads. The test system manages multiple nodes to be tested simultaneously, realizing a batch test communication test process: the test system issues test instructions and multi-node communication test programs to each node to be tested, and the nodes to be tested return test results after executing the test instructions.

[0166] Step 720: Based on the structured information, output the structured test results corresponding to the multi-node communication test.

[0167] The structured test results corresponding to multi-node communication tests are used to instruct testers to evaluate the communication capabilities between multiple nodes under test, enabling centralized management and reproducible analysis of the test results. Illustratively, the JSON file corresponding to the structured information is read and converted into a comma-separated values ​​(CSV) file as the structured test result; alternatively, a visual chart is generated based on the CSV file as the structured test result.

[0168] Among them, the visualization charts are used to graphically display the performance data corresponding to the multi-node communication test between multiple nodes under test and the structured test results corresponding to the pre-test. Optionally, the visualization charts can be implemented as a web dashboard, which provides a graphical interactive interface for real-time indicator data visualization, status alarm information and resource management of the test system in the communication test process through a web browser.

[0169] In some embodiments, the structured test results include prompts regarding the performance data. Reference performance data is obtained, which indicates the theoretical value of the performance data; the reference performance data is compared with the performance data to obtain a performance comparison result, which is included in the structured test results.

[0170] Schematic illustration: The testing system prints performance data obtained from multi-node communication tests based on structured information. If the performance data is small compared to the reference performance data, an alarm message may be generated as a structured test result, indicating to the testers that there may be communication problems between the multiple nodes under test. If the performance data is large compared to the reference performance data, a structured test result indicating a successful test is generated. The structured test results, including key structured test results for multi-node communication tests such as alarm messages or test success notifications, provide testers with a clear view of the test process, enabling them to further adjust the communication system corresponding to the multiple nodes under test and improve the efficiency of the testing process.

[0171] Optionally, the aforementioned reference performance data includes, but is not limited to, performance data peaks pre-set by testers based on theory, performance data baselines set within historical time periods, or reference data indicated by industry standards. By comparing the reference performance with the performance data actually measured during the testing process, structured test results are obtained, subjective judgments are transformed into objective and quantitative evaluation conclusions, avoiding reliance on testers' personal experience to arrive at vague and individualized evaluation conclusions, eliminating subjective bias, and improving the efficiency of obtaining structured test results.

[0172] In some embodiments, structured information is analyzed using a large language model to generate and output structured test results. The large language model is pre-trained using a reference dataset, which includes, but is not limited to, structured information obtained through communication testing processes over historical time periods and the corresponding structured test results, or structured information obtained from an open-source platform and the corresponding structured test results.

[0173] By simultaneously processing multi-dimensional communication capability metrics such as bandwidth, latency, and packet loss rate using a large language model, complex communication problems that are difficult to detect with manually set rules can be identified. For example, high bandwidth accompanied by high jitter corresponds to communication congestion between multiple test nodes. This serves as a reference for evaluating the communication capabilities of heterogeneous machine learning platform clusters, improving the efficiency of testers in obtaining information from structured data. Furthermore, the large language model can dynamically adjust the judgment benchmark based on test configuration information, test load, and other test-related parameters, or achieve trend prediction of performance data, further improving the efficiency of obtaining structured test results.

[0174] In summary, the method provided in this application constructs a unified test environment to perform configuration processing on multiple nodes to be tested. Pre-tests and multi-node communication tests are automatically executed sequentially within this test environment. The pre-test serves as a preliminary check for multi-node communication testing. Once the pre-test passes (i.e., the test node's network is connected and resources are available), communication testing is performed with complete test processing logic to obtain performance data. This eliminates the need for testers to manually configure test parameters or execute test procedures, reducing repetitive debugging costs, manpower costs, and configuration time, thereby improving communication testing efficiency. Furthermore, it decouples communication problems on a single test node indicated by the pre-test from communication problems between at least two test nodes indicated by the multi-node communication test, enabling the location of communication anomalies to a single test node and improving the accuracy of communication testing.

[0175] By collecting test results from each test stage in the communication testing process and automatically summarizing the results through the testing system, structured information is generated as the test results for a unified communication testing process. This makes the test results and performance data comparable and traceable, which is beneficial for analyzing the communication capabilities between multiple nodes under test or achieving long-term performance comparisons. It also provides a standardized data foundation for performance analysis of heterogeneous machine learning platforms corresponding to multiple nodes under test. The structured test results obtained through preset rules can intuitively demonstrate the communication capabilities between multiple nodes under test, improving the efficiency of testers in obtaining information from structured data.

[0176] For illustrative purposes, please refer to the following: Figure 8 This illustrates a schematic diagram of the structure of a test system execution module provided in an exemplary embodiment of this application. Figure 8 As shown, the testing system includes at least seven execution modules. Each execution module is a component responsible for controlling, coordinating, and managing specific functions, and can be implemented through a manager. The testing system receives test requests created by testers for the heterogeneous machine learning platform cluster under test and builds the test environment. Through the configuration information acquisition module 810, it receives the test request, which includes a unified matching file, parses it, and obtains structured test configuration information. Optionally, the task scheduling and control module 820 creates container instances on each node under test based on the test configuration information to provide the test environment.

[0177] like Figure 8As shown, the configuration information acquisition module 810 transmits the test configuration information to the task scheduling and control module 820. The task scheduling and control module 820 generates and manages at least one test task in the communication test process and distributes the test tasks to the corresponding execution modules and nodes to be tested for execution. Optionally, the fixed task generation and allocation logic in the task scheduling and control module 820 is pre-set. Based on the test configuration information, at least one test task is generated in the communication test process through the above-mentioned fixed logic.

[0178] like Figure 8 As shown, the task scheduling and control module 820, network connectivity testing module 830, resource availability testing module 840, and node-to-node communication testing module 850 perform pre-tests on multiple nodes to be tested in the test environment. Based on the test configuration information, the task scheduling and control module 820 sends a list of nodes to be tested to the network connectivity testing module 830. The network connectivity testing module 830 performs a network connectivity test on any two nodes from the list of nodes to be tested, targeting the basic network. If the network connectivity test fails, the task scheduling and control module 820 receives the test failure information sent by the network connectivity testing module 830 and terminates the subsequent communication test process. Regardless of whether the test fails or not, the task scheduling and control module 820 records the test results of the network connectivity test to the conclusion output module 860.

[0179] like Figure 8 As shown, if the network connectivity test passes, the task scheduling and control module 820 sends preset node configuration data to the resource availability test module 840 based on the test configuration information. Based on the node configuration data, the resource availability test module 840 checks the hardware and software status of each node under test to ensure that the operating environment of the node under test meets the requirements of multi-node communication testing. If the resource availability test fails, the task scheduling and control module 820 receives the test failure information sent by the resource availability test module 840 and terminates the subsequent communication test process. Regardless of whether the test fails or not, the task scheduling and control module 820 records the test results of the resource availability test to the conclusion output module 860.

[0180] like Figure 8As shown, if the resource availability test passes, the task scheduling and control module 820, based on the test results corresponding to the network connectivity test and resource availability test, selects the nodes to be tested that have passed the above two-stage pre-test as valid nodes, and sends the valid node information to the node-to-node communication test module 850. The node-to-node communication test module 850 performs a node-to-node communication test on any one of the at least two valid nodes, and obtains the node-to-node communication test result. The node-to-node communication test result is used to indicate whether the communication capability between any two nodes to be tested meets the requirements of the multi-node communication test. If the node-to-node communication test fails, the task scheduling and control module 820 receives the test failure information sent by the node-to-node communication test module 850 and terminates the subsequent communication test process; regardless of whether the test fails or not, the task scheduling and control module 820 records the test result of the node-to-node communication test to the conclusion output module 860.

[0181] like Figure 8 As shown, the multi-node communication test module 870, after passing the pre-test, performs multi-node communication tests on multiple nodes under test in the test environment to obtain performance data. The task scheduling and control module 820 records the performance data of the multi-node communication test to the conclusion output module 860. The conclusion output module 860 uniformly organizes and processes at least one of the pre-test results and performance data to obtain structured information, and obtains structured test results based on the structured information. The structured information includes communication problem information and corresponding test stage information in the event of communication problems in any test stage of the communication test process. Optionally, obtaining structured test results based on structured information includes outputting them to testers in a visual form to evaluate the communication capabilities of the heterogeneous machine learning platform cluster corresponding to the multiple nodes under test.

[0182] Figure 9 This is a structural block diagram of a communication testing apparatus provided in an exemplary embodiment of this application, as shown below. Figure 9 As shown, the device includes the following parts: Environment building module 910 is configured to build a test environment, which includes multiple nodes to be tested. The pre-test module 920 is configured to perform pre-tests on multiple nodes under test in a test environment. The pre-tests are used to check the network connectivity and resource availability of the nodes under test. The communication test module 930 is configured to perform multi-node communication tests on multiple nodes under test if the pre-test passes, and obtain performance data. The performance data is used to characterize the communication performance between multiple nodes under test.

[0183] In an optional embodiment, the pre-test module 920 is further configured to perform network connectivity tests on any two of the plurality of nodes to be tested. The pre-test module 920 is also configured to perform resource availability tests on multiple nodes to be tested. The pre-test module 920 is also configured to perform a node-to-node communication test on any two nodes under test if the network connectivity test and resource availability test are passed, and obtain the node-to-node communication test result. The node-to-node communication test result is used to indicate whether the communication capability between any two nodes under test meets the requirements of the multi-node communication test.

[0184] In one optional embodiment, the number of nodes to be tested is N, where N is an integer greater than 1; The pre-test module 920 is also configured to perform network connectivity tests on any two of the multiple test nodes, including: The pre-test module 920 is also configured to send test data to N-1 other nodes besides itself based on each node under test. The test data is used to test the availability of the communication link between the node under test and other nodes.

[0185] In an optional embodiment, the pre-test module 920 performs resource availability tests on multiple nodes to be tested, including one or more of the following: The pre-test module 920 is also configured to perform hardware verification processing on multiple nodes under test, which is used to check the hardware availability of the nodes under test. The pre-test module 920 is also configured to perform software verification processing on multiple nodes under test, which is used to check the driver availability of the nodes under test. The pre-test module 920 is also configured to perform single-node communication tests on multiple nodes under test. The single-node communication tests are used to check the compatibility between the hardware and drivers of the nodes under test.

[0186] In an optional embodiment, the pre-test module 920 is further configured to establish a communication link between any two nodes to be tested; The pre-test module 920 is also configured to obtain node pair performance parameters between any two nodes under test based on the communication link. The node pair performance parameters are used to characterize the network performance between any two nodes under test. The pre-test module 920 is also configured to obtain node pair communication test results based on node pair performance parameters. The node pair communication test results are used to indicate whether the node pair performance parameters have reached a preset network performance threshold.

[0187] In an optional embodiment, the environment building module 910 is further configured to acquire test configuration information, which describes the test environment; Environment building module 910 is also configured to build a test environment based on test configuration information.

[0188] In an optional embodiment, the environment building module 910 is further configured to create container instances corresponding to the test environment for multiple nodes to be tested based on the image information in the test configuration information. The container instances are used to provide an execution environment for pre-testing and multi-node communication testing.

[0189] In an optional embodiment, after the environment building module 910 obtains the test configuration information, the environment building module 910 is further configured to create a shared storage environment among multiple nodes to be tested based on the path information in the test configuration information. The shared storage environment is used to store the test configuration information, the test results corresponding to the pre-test, and the performance data.

[0190] In an optional embodiment, the communication test module 930 is further configured to obtain at least two valid nodes based on the test results corresponding to the pre-test, wherein the at least two valid nodes are nodes that have passed the pre-test among a plurality of nodes to be tested; The communication test module 930 is also configured to perform multi-node communication tests on at least two valid nodes to obtain performance data.

[0191] In an optional embodiment, Figure 10 This is a structural block diagram of a communication testing apparatus provided in another exemplary embodiment of this application; as shown below. Figure 10 As shown, the communication testing device also includes a result processing module 940, which is configured to perform data processing on at least one of the test results and performance data corresponding to the pre-test to obtain structured information. The results processing module 940 is also configured to output structured test results corresponding to multi-node communication tests based on structured information.

[0192] In an optional embodiment, the result processing module 940 is further configured to acquire reference performance data, which is used to indicate the theoretical value of the performance data. The results processing module 940 is also configured to compare the reference performance data with the performance data to obtain performance comparison results. The structured test results include the performance comparison results.

[0193] In an optional embodiment, the results processing module 940 is further configured to analyze the structured information using a large language model, generate structured test results, and output them.

[0194] In summary, the apparatus provided in this application, through the association between the business identifier, session identifier, and link identifier corresponding to the business request, can clearly identify all interfaces called when performing business functions based on the business request, thereby determining the complete interface call chain. Based on the interface call chain, the interface call logic corresponding to the business request can be clearly identified, thereby generating interface call code that can automatically call interfaces to perform business functions, and generating result checking code that can automatically check the database data generated when performing business functions. By concatenating the interface call code and the result checking code, automated test case code is obtained, enabling test cases to be automatically generated based on the automated test case code, automatically calling interfaces to perform corresponding business functions, and automatically checking the execution results, thus realizing the automatic testing of the results of performing business functions. This achieves the automated determination of the interface call chain and improves the generation efficiency of automated test cases.

[0195] It should be noted that the communication testing device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0196] Figure 11 A structural block diagram of a terminal 1100 provided in an exemplary embodiment of this application is shown. The terminal 1100 may be a smartphone, tablet computer, MP3 player, MP4 player, laptop computer, or desktop computer. The terminal 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0197] Typically, terminal 1100 includes a processor 1101 and a memory 1102.

[0198] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), FPGA, and Programmable Logic Array (PLA). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0199] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 is used to store at least one instruction, which is executed by the processor 1101 to implement the communication testing method provided in the method embodiments of this application.

[0200] In some embodiments, the terminal 1100 also includes other components 1103, as those skilled in the art will understand. Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0201] Embodiments of this application also provide a computer device that can be implemented as follows: Figure 1 The terminal or server shown. The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the communication testing methods provided in the above-described method embodiments.

[0202] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the communication testing method provided in the above-described method embodiments.

[0203] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the communication testing methods provided in the above-described method embodiments.

[0204] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0205] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0206] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A communication testing method, characterized in that, The method includes: Build a test environment, which includes multiple nodes to be tested; In the test environment, pre-tests are performed on the plurality of nodes to be tested, and the pre-tests are used to check the network connectivity and resource availability of the nodes to be tested. If the pre-test passes, a multi-node communication test is performed on the plurality of nodes to be tested to obtain performance data, which is used to characterize the communication performance between the plurality of nodes to be tested.

2. The method according to claim 1, characterized in that, The step of performing pre-tests on the plurality of nodes to be tested in the test environment includes: Perform a network connectivity test on any two of the plurality of nodes to be tested; Perform resource availability tests on each of the multiple nodes to be tested; If the network connectivity test and the resource availability test pass, a node-to-node communication test is performed on any two of the plurality of nodes to be tested to obtain the node-to-node communication test result. The node-to-node communication test result is used to indicate whether the communication capability between any two nodes to be tested meets the requirements of the multi-node communication test.

3. The method according to claim 2, characterized in that, The number of nodes to be tested is N, where N is an integer greater than 1; The step of performing a network connectivity test on any two of the plurality of nodes to be tested includes: Each node under test sends test data to N-1 other nodes under test, and the test data is used to test the availability of the communication link between the node under test and other nodes.

4. The method according to claim 2, characterized in that, The process of performing resource availability testing on the plurality of nodes to be tested includes one or more of the following: Hardware verification processing is performed on each of the plurality of nodes to be tested, and the hardware verification processing is used to check the hardware availability of the nodes to be tested. Software verification processing is performed on each of the plurality of nodes to be tested, and the software verification processing is used to check the driver availability of the nodes to be tested; Single-node communication tests are performed on each of the multiple nodes to be tested. The single-node communication tests are used to check the compatibility between the hardware and the driver of the node to be tested.

5. The method according to claim 2, characterized in that, If the network connectivity test and the resource availability test pass, a node-to-node communication test is performed on any two of the plurality of nodes to be tested to obtain the node-to-node communication test results, including: Establish a communication link between any two nodes to be tested; Based on the communication link, obtain the node pair performance parameters between any two nodes under test, and the node pair performance parameters are used to characterize the network performance between any two nodes under test. Based on the node pair performance parameters, the node pair communication test results are obtained, and the node pair communication test results are used to indicate whether the node pair performance parameters have reached a preset network performance threshold.

6. The method according to claim 1, characterized in that, The construction of the test environment includes: Obtain test configuration information, which describes the test environment; Based on the test configuration information, the test environment is constructed.

7. The method according to claim 6, characterized in that, The step of constructing the test environment based on the test configuration information includes: Based on the image information in the test configuration information, container instances corresponding to the test environment are created for each of the multiple nodes to be tested. The container instances are used to provide the execution environment for the pre-test and the multi-node communication test.

8. The method according to claim 6, characterized in that, The method further includes: Based on the path information in the test configuration information, a shared storage environment is created among the multiple nodes to be tested. The shared storage environment is used to store the test configuration information, the test results corresponding to the pre-test, and the performance data.

9. The method according to claim 1, characterized in that, If the pre-test passes, a multi-node communication test is performed on the multiple nodes to be tested to obtain performance data, including: Based on the test results corresponding to the pre-test, at least two valid nodes are obtained, wherein the at least two valid nodes are the nodes that pass the pre-test among the plurality of nodes to be tested; Perform multi-node communication tests on the at least two valid nodes to obtain the performance data.

10. The method according to claim 1, characterized in that, The method further includes: Perform data processing on at least one of the test results corresponding to the pre-test and the performance data to obtain structured information; Based on the structured information, the structured test results corresponding to the multi-node communication test are output.

11. The method according to claim 10, characterized in that, The method further includes: Obtain reference performance data, which is used to indicate the theoretical value of the performance data; The performance comparison results are obtained by comparing the reference performance data with the performance data, and the structured test results include the performance comparison results.

12. The method according to claim 10, characterized in that, The step of outputting the structured test results corresponding to the multi-node communication test based on the structured information includes: The structured information is analyzed using a large language model to generate and output the structured test results.

13. A communication testing device, characterized in that, The device includes: The environment building module is configured to build a test environment, which includes multiple nodes to be tested. The pre-test module is configured to perform pre-tests on the plurality of nodes to be tested in the test environment, wherein the pre-tests are used to check the network connectivity and resource availability of the nodes to be tested; The communication testing module is configured to perform multi-node communication tests on the plurality of nodes to be tested if the pre-test passes, and obtain performance data, which is used to characterize the communication performance between the plurality of nodes to be tested.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the communication testing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the communication testing method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the communication testing method as described in any one of claims 1 to 12.

Images