Network automation testing methods, devices, equipment and media

Abstract

This invention discloses a network automation testing method, apparatus, device, and medium. The method includes: acquiring configuration files and test cases; automatically generating a list of communication pairs required for testing based on node information and network scenario type; establishing and managing connections with all nodes according to the node information in the communication pair list; automatically executing performance test commands sequentially on the corresponding client and server nodes to generate network traffic and acquire raw test output; automatically extracting performance indicators from the raw test output, parsing the performance indicators, and generating a structured test report. This invention integrates configuration input, communication pair generation, remote connection management, automated performance test execution, data acquisition, and report generation into a closed-loop automated process. Users only need to provide configuration files and test cases to complete all testing work from environment preparation to result output with one click, improving test deployment efficiency and reusability.

Description

Technical Field

[0001] This invention relates to the field of network testing technology, and in particular to an automated network testing method, apparatus, equipment, and medium. Background Technology

[0002] Network testing is an important means of functional verification and performance evaluation of communication protocols, device performance, and link behavior in computer network systems. Traditional network testing practices typically employ physical device-based testing platforms, manually configuring and scripting test tasks to evaluate system behavior under different loads. Current main application scenarios for network testing platforms include protocol verification, device selection, network topology evaluation, and algorithm optimization.

[0003] RDMA (Remote Direct Memory Access) is a high-performance network transmission technology that achieves end-to-end low-latency, high-bandwidth data transmission through zero-copy and kernel bypass mechanisms. It is widely used in scenarios such as distributed storage in data centers and high-performance computing. Its core advantage lies in bypassing the operating system kernel to directly access application memory, reducing CPU intervention, making it particularly suitable for latency-sensitive business scenarios. The perftest testing component is a mainstream toolset for RDMA network testing, supporting RDMA technologies such as InfiniBand, RoCE, and iWARP, and providing basic performance testing capabilities such as bandwidth (e.g., ib_write_bw) and latency (e.g., ib_write_lat).

[0004] Existing RDMA networking scenario tests have the following shortcomings: 1. Manual testing process leads to low efficiency: Current testing relies on manual execution of commands, coordination of nodes, and aggregation of results. When testing multiple nodes and scenarios, the workload increases exponentially. Taking a 10-node All-to-All scenario as an example, it requires manually executing 100 independent commands and configuring 200+ parameters. Furthermore, the test environment configuration cannot be reused, resulting in long testing cycles and high labor costs.

[0005] 2. Limited testing dimensions and lack of structured evaluation: Tools like perftest only support basic metrics such as bandwidth and latency, failing to cover key evaluation dimensions of data center networks. For example, they cannot monitor congestion control (PFC trigger threshold, ECN (Explicit Congestion Notification) marking ratio), traffic fairness (lack of analysis on the balance of bandwidth allocation across links during multi-stream contention), and dynamic fluctuations (unable to record time-series data such as bandwidth jitter and latency distribution). Furthermore, test results are output as unstructured text logs, requiring manual parsing and cannot be directly used for trend analysis or anomaly detection.

[0006] 3. Poor Flexibility of Test Load / Business Model: Existing tools suffer from cumbersome parameter configuration and high scenario switching costs when switching between different business scenarios. Specifically, this manifests as: Complex Multi-Scenario Switching: Taking the switch from a "two-to-one" scenario (2 sending nodes → 1 receiving node, 1 QP, 1M message size) to an "All-to-All" scenario (3 nodes fully interconnected, 8 QP, 256KB message size) as an example, the following steps are required: Log in to all nodes and terminate the current test process; Modify the number of QPs (1→8) and message size (1M→256KB) in the configuration file of each node; Re-establish the connection relationship between nodes (from star topology to mesh topology); Manually coordinate the synchronous startup of all nodes, a process that takes more than 30 minutes. This results in low scenario reusability, and the test configurations for different business scenarios cannot be standardized and stored, requiring the parameter document to be rewritten for each test. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide a network automation testing method, apparatus, equipment and medium that overcomes or at least partially solves the above problems.

[0008] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0009] According to a first aspect of the present invention, a network automation testing method is provided, comprising: Obtain the configuration file and test cases. The configuration file contains node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. Based on the node information and the network scenario type, a communication pair list required for testing is automatically generated. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. Based on the node information in the communication pair list, establish and manage connections with all nodes; Based on the communication pair list, test parameters, and established connections, performance test commands are automatically executed sequentially on the corresponding client and server nodes to generate network traffic and obtain raw test output. Performance metrics are automatically extracted from the original test output, the performance metrics are parsed, and a structured test report is generated.

[0010] According to a first aspect of the present invention, a network automation testing device is provided, the device comprising: The load generation module is used to obtain configuration files and test cases. The configuration files contain node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. The communication pair list generation module is used to automatically generate a communication pair list required for testing based on the node information and the network scenario type. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. The connection management module is used to establish and manage connections with all nodes based on the node information in the communication pair list; An automated test execution module is used to automatically execute performance test commands sequentially on the corresponding client nodes and server nodes based on the communication pair list, test parameters, and the established connections, so as to generate network traffic and obtain raw test output. The data acquisition and analysis module is used to automatically extract performance indicators from the original test output, analyze the performance indicators, and generate a structured test report.

[0011] According to a third aspect of the present invention, a computer device is provided, including a processor and a memory, the memory storing computer program instructions executable by the processor, wherein when the processor executes the computer program instructions, it implements the instructions as described in any of the above methods.

[0012] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein computer program instructions are stored therein, the computer program instructions being loaded and executed by a processor to perform the operations performed by the method described in any of the preceding claims.

[0013] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: This invention provides a network automation testing method, apparatus, device, and medium. The method integrates configuration input, communication pair generation, remote connection management, automated performance testing, data acquisition, and report generation into a closed-loop automated process. Users only need to provide configuration files and test cases to complete all testing work from environment preparation to result output with one click. At the same time, the unified encapsulation of remote node control logic realizes automated operations for batch node initialization, remote command execution, and status reading based on SSHConnection, replacing the inefficient process of traditional manual login and individual script execution, and improving test deployment efficiency and reusability.

[0014] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an automated network testing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the principle structure of a network automation testing device provided in an embodiment of the present invention; Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings.

[0018] The accompanying drawings illustrate various structural schematics according to embodiments of this application. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0019] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. In the context of this application, similar or identical parts may be represented by the same or similar reference numerals.

[0020] To better understand the above technical solutions, the following will describe the above technical solutions in detail with reference to specific implementation methods. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0021] Figure 1 This is a flowchart illustrating an automated network testing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this automated network testing method includes the following steps: S1. Obtain the configuration file and test cases. The configuration file contains node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. In this embodiment of the invention, the configuration file contains an abstract description of the test environment (i.e., node information of the test environment). The node information of the test environment includes, for example, node mapping relationships, node IPs, network card information, test parameters (number of QPs (Queue Pairs), message size, runtime), login username and password (or key), etc. The node information configuration items are loaded in a structured form to ensure that the node information is decoupled from the identity authentication logic, which facilitates batch management. The test cases contain the network scenario type to be executed and the corresponding test parameters. The network scenario type includes, for example, Incast mode, Outcast mode, or All-to-All mode. The test parameters are, for example, combinations of the corresponding test parameters in the configuration file. For example, a test case is a 4-node Incast scenario, where 3 nodes communicate with another node, execute writeonly operations, and the number of QPs to be tested is 1, 2, or 4, with a message size of 1kb to 16kb. By pre-defining a set of environment configuration files and then creating multiple test case files to cover different test scenarios (such as stress testing, long-term stability testing, and extreme testing), when a test task needs to switch scenarios, it is only necessary to load different configuration files or test cases without manually modifying the parameters of each node.

[0022] S2. Based on the node information and the network scenario type, automatically generate a communication pair list required for testing. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. After obtaining the configuration file and the test cases, this embodiment of the invention automatically generates a list of communication pairs required for testing based on the node information and the network scenario type. This includes: confirming the network scenario type in the test cases, where the network scenario type includes Incast mode, Outcast mode, or All-to-All mode; obtaining a node list based on the node information; and establishing node mapping relationships for all nodes in the node list using the communication pair generation strategy corresponding to the network scenario type, thereby automatically generating the list of communication pairs required for testing. Specifically, the Incast mode is used to generate communication pairs between multiple client nodes and one server node, the Outcast mode is used to generate communication pairs between one client node and multiple server nodes, and the All-to-All mode is used to generate all pairs of interconnected communication pairs between nodes.

[0023] The communication pair generation strategy includes, for example, topology interpretation, communication pair generation, and node mapping. Topology interpretation is used to parse the topology specified in the test cases and further complete the node mapping relationships. In other embodiments of the invention, when the number of nodes specified by the user is insufficient to form a complete topology, the node list (client nodes or server nodes) can be automatically expanded through node reuse to ensure the complete operation of the test task. Communication pair generation is used to automatically calculate and output a one-to-one correspondence list between client nodes and server nodes based on the node list defined in the configuration file and the scenario identifier specified by the user in the test cases (i.e., the confirmed network scenario type, including Incast mode, Outcast mode, or All-to-All mode), generating the original pairing relationships. Node mapping is used to finally determine and output the role mapping result of each test node in a specific task based on the structural information provided by topology interpretation and the generated original pairing relationships, thus ultimately forming a complete (test-required) communication pair list that can be used for downstream execution.

[0024] S3. Based on the node information in the communication pair list, establish and manage connections with all nodes; After generating the communication pair list, connections with all nodes are established and managed. These connections may be, for example, SSH sessions. In this embodiment of the invention, establishing and managing connections with all nodes based on the node information in the communication pair list includes: reading and parsing the node information; concurrently logging into each node via SSH and verifying the node information; establishing connections with all verified nodes and adding the connections to an active connection pool; sending target shell commands to the nodes via a remote command execution interface and receiving the command output results; and managing the connections based on the command output results.

[0025] Specifically, the embodiments of the present invention include an SSH connection pool manager, an authenticator, and a remote command scheduler. The SSH connection pool manager instantiates a corresponding SSHConnection object (encapsulated based on the paramiko library) for each node based on the parsed node information (a list of client and server nodes), and establishes a stable remote SSH session. During the establishment process, an exception handling mechanism is encapsulated to handle errors such as authentication failure, network interruption, and unreachable target nodes, and accurately records them in the log. The authenticator verifies the username, password, or key during login to ensure subsequent operation permissions. The remote command scheduler manages the command execution process uniformly on top of the connection pool, including executing performance testing tools such as ib_write_bw and ib_write_lat on specified nodes, or executing devmem read / write register operations.

[0026] This invention employs an SSH connection pool manager to perform concurrent SSH logins to all nodes and maintains an active connection pool. This connection pool supports connection reuse in a multi-threaded environment, avoiding the need to repeatedly establish connections for each command and improving execution efficiency. The remote command execution interface (e.g., the standard interface `exec_command()`) can send arbitrary shell commands to remote nodes and receive their standard output and standard error output. This process can be uniformly managed by the remote command scheduler. The remote command scheduler includes executing performance testing tools such as `ib_write_bw` and `ib_write_lat` on specified nodes, or performing `devmem` register read / write operations. It supports command execution timeout control and returns a packaged `SSHCommandResult` object, providing a unified format for data input to upper-layer modules. Furthermore, after testing is completed (or an exception occurs requiring interruption), the `close()` method is called to actively disconnect the connection and release system resources.

[0027] S4. Based on the communication pair list, test parameters, and the established connection, automatically execute performance test commands sequentially on the corresponding client node and server node to generate network traffic and obtain raw test output; After the connection is established, this embodiment of the invention loads a combination of multiple sets of test parameters in the test case, generates a specific execution task according to the communication pair list, and starts the server process on all server nodes in sequence through a multi-threaded concurrency mechanism, and then starts the client process on all client nodes to initiate performance test traffic to the specified server (such as using ib_write_bw or ib_write_lat).

[0028] Specifically, based on the communication pair list, test parameters, and established connections, this embodiment of the invention automatically executes performance test commands sequentially on corresponding client nodes and server nodes to generate network traffic and obtain raw test output. This includes: loading the test cases and extracting multiple combinations of the test parameters; for each combination of test parameters, generating a corresponding traffic initiation command according to the communication pair list, and concurrently starting server processes on all server nodes according to the traffic initiation command; after the server processes start, generating performance test commands on all corresponding client nodes, starting client processes on the corresponding client nodes according to the performance test commands, and initiating performance test traffic to the specified server node to obtain the raw test output.

[0029] After reading and parsing the node information and initializing the corresponding SSHConnection instance, multiple sets of test parameter combinations are extracted. A streaming start command (including configurations such as specified port number, message size, MTU size, and QP quantity) is constructed on all server nodes for each set of test parameter combinations. This streaming start command concurrently starts server processes on all server nodes, generating a specific execution task for each client-server pair and determining the task execution order and concurrency strategy. After the server processes are ready, a performance test command is constructed on all client nodes with the same parameters, specifying the corresponding server's IP address as the target address. The command is then executed again via SSH to initiate a connection, thereby starting client processes on the corresponding client nodes according to the performance test command and sending performance test traffic to the specified server nodes to obtain the original test output. The execution of all performance test commands is implemented through a unified interface, perftest(), and supports wait and sleep_time parameters to control whether to wait for a return and to delay for a certain number of seconds after startup to ensure the server is ready.

[0030] This invention provides two modes for initiating performance test traffic to the specified server node: limited-duration streaming (e.g., -D 10 indicates streaming for 10 seconds) and unlimited-duration streaming (e.g., --run_infinitely combined with -D 1 indicates printing performance data once per second). In unlimited mode, a background thread can be started to continuously read standard output, and the perftest_run_infinitely() method returns the real-time output lines during the streaming process as an iterator. This invention employs a multi-threaded concurrency control mechanism (based on ThreadPoolExecutor) for client connection and execution processes, ensuring concurrent execution of multi-node test tasks. After the streaming is completed, the close() interface is called to uniformly close all SSH sessions and release system resources. When initializing a Scale instance, test parameters such as s (message size), m (runtime), and q (number of QPs) are injected into the test configuration. Users can define multiple combinations of test parameters at once for batch testing (such as enumerating combinations of different QP numbers and message sizes in run_test()). Testers only need to call interfaces such as perftest(), without needing to care about the underlying command details and SSH interaction logic. The details of command concatenation are hidden, and a standardized execution interface is uniformly exposed to the outside world. Testers only need to specify basic parameters to complete complex test processes with one click.

[0031] S5. Automatically extract performance indicators from the original test output, parse the performance indicators, and generate a structured test report.

[0032] This invention embodiment automatically extracts performance metrics from the original test output, parses the performance metrics, and generates a structured test report, including: parsing basic performance metrics from the original test output, the basic performance metrics including at least average bandwidth; and calculating derived performance metrics for evaluating network status based on the basic performance metrics and generating a structured test report, the derived performance metrics including at least one of bandwidth utilization, bandwidth balance, and bandwidth jitter.

[0033] The basic performance metrics are determined based on the test scenario. For example, in a bandwidth test scenario, this embodiment of the invention obtains the original test output (the standard output returned after the client SSHConnection executes the streaming command) by initiating performance test traffic to the specified server node, extracts bandwidth-related metrics through regular expression matching, and encapsulates them in get_bw(). The extracted basic performance metrics in the method include: number of bytes sent (used to calculate the total amount of data actually transmitted during the test, which can be used to verify whether the test duration and bandwidth match, or to detect whether packet loss leads to insufficient transmission), number of iterations (reflecting the number of execution rounds within the testing tool, which can help determine whether the test has been completed, or to observe stability during long-term testing), instantaneous peak bandwidth (bw_peak, used to evaluate the maximum carrying capacity of the network under short-term burst traffic, to determine whether there is instantaneous congestion or hardware bottleneck, and compared with the average bandwidth, it can analyze the bandwidth fluctuation), average bandwidth (bw_average), and message rate (msg_rate, representing the number of message packets transmitted per second, not the number of bits, which can be used to measure the completion rate of RDMA operations in small message scenarios, reflecting the processing capabilities of the CPU and network card), etc.; in the latency test scenario, this embodiment of the invention uses get_lat() The method parses the original test output and extracts the basic performance indicators, including minimum latency (reflecting the network's optimal performance under ideal conditions (no congestion, no contention), which can be used as a theoretical lower limit reference for system optimization), maximum latency (identifying whether there is severe abnormal latency (such as due to congestion, packet loss and retransmission, GC pauses, etc.), used to detect long-tail problems), typical value (such as mode or median, representing the most common latency performance, which reflects the user experience under normal conditions better than the average value, avoiding being skewed by extreme values), average value, standard deviation (measuring the degree of latency dispersion, the larger the standard deviation, the more severe the latency jitter), as well as P99 latency and P99.9 latency, etc.

[0034] This invention can also collect key hardware feedback metrics in the network, including the receive PFC count (get_rxpfc() interface) at the transmitting end, the transmit PFC count (get_txpfc() interface) at the receiving end, and the cumulative ECN tag value (get_ecn() interface) at the receiving end. This process is implemented by reading the count value on the node through the register-level interface devmem(). The difference is sampled and calculated before and after the performance test to quantify the congestion control events that occur during the test. The changes in the hardware feedback metrics characterize the PFC triggering frequency and ECN congestion tagging behavior in the network and can be used to analyze the effectiveness of the congestion control mechanism.

[0035] In long-duration streaming scenarios, for the original test output of each round of streaming, this embodiment of the invention calculates derived performance indicators for evaluating network status based on the basic performance indicators. The derived performance indicators include bandwidth utilization, bandwidth balance, and bandwidth jitter. The larger the bandwidth utilization value, the more fully the link is utilized. The closer the bandwidth balance value is to 1, the more balanced the bandwidth distribution among the links. The bandwidth jitter is the maximum jitter value among all communication pairs. The smaller the bandwidth jitter value, the more stable the streaming process.

[0036] The bandwidth utilization rate is obtained based on the following formula: ; In the formula, U represents bandwidth utilization, and ; Characterizes the average bandwidth of all communication pairs; Characterizes the upper limit of theoretical bandwidth.

[0037] The bandwidth equalization is obtained based on the following formula: ; In the formula, B represents bandwidth balance, and ; and These represent the maximum and minimum average bandwidth values ​​among all communication pairs, respectively.

[0038] The bandwidth jitter is obtained based on the following formula: ; ; In the formula, Characterizes the jitter of a single communication pair; The set of average bandwidths obtained from the N most recent raw test outputs; Total number of communication pairs; Characterizes bandwidth jitter.

[0039] After extracting performance metrics, this embodiment of the invention parses the performance metrics and generates a structured test report, which is written to an Excel file using pandas (via the write_to_excel() interface). The fields include the previously extracted data such as traffic type, parameter configuration, bandwidth utilization, bandwidth balance, bandwidth jitter, latency statistics, PFC count, ECN count, etc.

[0040] In this embodiment of the invention, the execution process, key events and running results are fully recorded, grouped and categorized and sorted in a structured manner in the automated testing process. This embodiment of the invention automatically records the acquired configuration files, test cases, original test outputs and performance indicators, and generates log information. Furthermore, the log information is grouped and sorted according to the communication pair identifier to form a structured log file with timestamps.

[0041] Specifically, this embodiment of the invention uses pytest's hook functions and user-defined sorting logic as its core implementation, combining console and file output mechanisms. During the automated test initialization phase, the pytest_configure() method creates a timestamp-named log file, ensuring each test round has independent log recording space. The log file output includes the test name, start and end times, streaming parameters (such as message size and QP count), communication pair bandwidth / latency statistics, PFC / ECN metric reading results, command execution status, etc., using a unified output format for easy later metric analysis. During automated testing, all log information is simultaneously output to the console and log file. Users can use the command-line parameter --nologs to disable console output, retaining only file records. After the automated test ends, this embodiment automatically calls pytest_sessionfinish() to trigger the log sorting function. The sorting logic is implemented by the sort_logs() function, which internally groups logs by identifying communication pair identifiers (such as get_bw [node-A]-[node-B]) and sorts them based on the port number or sequence number of the communication target node. All test cases are automatically inserted with a uniform start identifier (e.g., "Starting test:") before execution.<test_id> This facilitates quick location of log segments in multiple test scenarios. This embodiment of the invention silences the output of the paramiko library to avoid unnecessary low-level communication logs interfering with the main log content.

[0042] The automated network testing method described in this embodiment of the invention has the following advantages compared to existing technologies: 1. The configuration input, communication pair generation, remote connection management, automated performance testing, data collection and report generation are integrated into a closed-loop automated process. Users only need to provide configuration files and test cases to complete all testing work from environment preparation to result output with one click. At the same time, the unified encapsulation of remote node control logic realizes the automation of batch node initialization, remote command execution and status reading based on SSHConnection, replacing the inefficient process of traditional manual login and individual script execution, improving test deployment efficiency and reusability. 2. Based on scenario types such as Incast, Outcast, and All-to-All, the system automatically generates a list of communication pairs between the client and the server, and has the ability to reuse nodes and automatically complete them. Users only need to specify the nodes and communication modes, and the system can automatically generate communication pairs and organize streaming scheduling tasks. Test parameters such as message size, number of connections (QP), and runtime can be flexibly configured and automatically iterated. This technology effectively solves the problems of complex scenario construction and too many parameter combinations in traditional network simulation processes, and significantly improves the adaptability of the test system to large-scale and multi-dimensional experiments. 3. The automated test execution module adopts multi-threaded concurrency control. It first starts the process on all servers, and then initiates traffic concurrently on all clients. It supports batch traversal execution of parameter combinations. The concurrent scheduling mechanism can ensure the concurrent execution of multi-node test tasks and control the node start time difference to the millisecond level, thereby obtaining more accurate performance indicators such as congestion and fairness. It significantly reduces the time cost of test preparation and improves the controllability and repeatability of the test process. 4. Introduce a structured extraction mechanism for multi-dimensional performance indicators, and combine it with the log module to achieve automatic recording and grouping of the entire testing process, key events and running results. By parsing the performance indicators and tracking the results at the granular level of communication through log files, users can quickly locate performance bottlenecks or abnormal behaviors, thereby greatly improving the ability to analyze network testing problems.

[0043] Based on the above embodiments, as a supplement to the above... Figure 1 The present invention provides an embodiment of a network automated testing device to implement the method shown. This embodiment of the device is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices, see reference. Figure 2 As shown, the network automation testing device includes: The load generation module 100 is used to obtain configuration files and test cases. The configuration files contain node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. The communication pair list generation module 200 is used to automatically generate a communication pair list required for testing based on the node information and the network scenario type. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. The connection management module 300 is used to establish and manage connections with all nodes based on the node information in the communication pair list; The automated test execution module 400 is used to automatically execute performance test commands sequentially on the corresponding client node and server node based on the communication pair list, test parameters and the established connection, so as to generate network traffic and obtain raw test output. The data acquisition and analysis module 500 is used to automatically extract performance indicators from the original test output, analyze the performance indicators, and generate a structured test report.

[0044] Each module in the aforementioned automated network testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0045] The automated network testing device described in this embodiment can execute the automated network testing method provided in the above embodiments. The automated network testing device has the corresponding functional steps and beneficial effects of the automated network testing method described in the above embodiments. For details, please refer to the embodiments of the automated network testing method described above. The embodiments of this invention will not be repeated here.

[0046] In other embodiments of the present invention, the corresponding functional modules of the network automation testing device can be configured in the form of a graphical user interface (GUI), thereby reducing the reliance on command line operations and improving the user experience; the configuration and layout of the graphical user interface can be determined according to actual application requirements, and the embodiments of the present invention do not limit this.

[0047] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface, such as a network interface card, is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. The display unit is used to create a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0048] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0049] In one exemplary embodiment, a chip is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0050] In one exemplary embodiment, a network interface card is provided, including a chip as described in any of the above embodiments and multiple interfaces, wherein the chip communicates externally through the interfaces.

[0051] In one embodiment, a computer device is also provided, including a processor, a chip in any of the above embodiments, or a network interface card in any of the above embodiments, wherein the chip or the network interface card is used to schedule packets to the processor or the chip or the network interface card itself for processing, and the processor is used to process the packets scheduled by the chip or the network interface card.

[0052] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0053] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0054] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application may include at least one of relational databases and non-relational databases. Non-relational databases may include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited thereto.

[0055] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0056] Similarly, it should be understood that, for the purpose of simplification and aiding understanding of one or more aspects of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention above. Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and it should be noted that the above embodiments are illustrative of the invention and not restrictive, and that alternative embodiments can be devised by those skilled in the art without departing from its scope.

Claims

1. A network automation testing method, characterized in that, Includes the following steps: Obtain the configuration file and test cases. The configuration file contains node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. Based on the node information and the network scenario type, a communication pair list required for testing is automatically generated. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. Based on the node information in the communication pair list, establish and manage connections with all nodes; Based on the communication pair list, test parameters, and established connections, performance test commands are automatically executed sequentially on the corresponding client and server nodes to generate network traffic and obtain raw test output. The performance metrics are automatically extracted from the original test output, the performance metrics are parsed, and a structured test report is generated. The automatic generation of the communication pair list required for testing based on the node information and the network scenario type includes: Confirm the network scenario type in the test cases, which includes Incast mode, Outcast mode, or All-to-All mode; Obtain a node list based on the node information; Using the communication pair generation strategy corresponding to the network scenario type, a node mapping relationship is established for all nodes in the node list, and the communication pair list required for testing is automatically generated. The Incast mode is used to generate communication pairs between multiple client nodes and one server node, the Outcast mode is used to generate communication pairs between one client node and multiple server nodes, and the All-to-All mode is used to generate communication pairs that are interconnected between all nodes.

2. The automated network testing method according to claim 1, characterized in that, The step of establishing and managing connections with all nodes based on the node information in the communication pair list includes: Read and parse the node information, and perform SSH login and verify the node information for each node concurrently; Establish connections with all verified nodes and add the connections to the active connection pool; The system sends target shell commands to the node via a remote command execution interface and receives the command output results, and then manages the connection based on the command output results.

3. The automated network testing method according to claim 1, characterized in that, The step of automatically executing performance test commands sequentially on the corresponding client and server nodes based on the communication pair list, test parameters, and established connections to generate network traffic and obtain raw test output includes: Load the test cases and extract combinations of multiple sets of test parameters; For each combination of the test parameters, a corresponding streaming start command is generated based on the communication pair list, and the server process is started concurrently on all server nodes according to the streaming start command. After the server process starts, a performance test command is generated on all corresponding client nodes. Based on the performance test command, a client process is started on the corresponding client node, and performance test traffic is sent to the specified server node to obtain the original test output.

4. The automated network testing method according to claim 1, characterized in that, The step of automatically extracting performance metrics from the original test output, parsing the performance metrics, and generating a structured test report includes: Basic performance metrics are parsed from the original test output, and the basic performance metrics include at least the average bandwidth; Based on the aforementioned basic performance metrics, derived performance metrics for evaluating network status are calculated and a structured test report is generated. These derived performance metrics include at least one of bandwidth utilization, bandwidth balance, and bandwidth jitter.

5. The automated network testing method according to claim 4, characterized in that, The bandwidth utilization rate is obtained based on the following formula: ； In the formula, U represents bandwidth utilization, and ; Characterizes the average bandwidth of all communication pairs; Characterizes the upper limit of theoretical bandwidth.

6. The network automation testing method according to claim 4, characterized in that, The bandwidth equalization is obtained based on the following formula: ； In the formula, B represents bandwidth balance, and ; and These represent the maximum and minimum average bandwidth values ​​among all communication pairs, respectively.

7. The network automation testing method according to claim 4, characterized in that, The bandwidth jitter is obtained based on the following formula: ； ； In the formula, Characterizes the jitter of a single communication pair; The set of average bandwidths obtained from the N most recent raw test outputs; Total number of communication pairs; Characterizes bandwidth jitter.

8. The automated network testing method according to claim 1, characterized in that, The method further includes: It automatically records the acquired configuration files, test cases, raw test outputs, and performance metrics, and generates log information. The log information is grouped and sorted according to the communication pair identifier to form a structured log file with timestamps.

9. A network automation testing device, applied to the method described in any one of claims 1-8, characterized in that, The device includes: The load generation module is used to obtain configuration files and test cases. The configuration files contain node information of the test environment, and the test cases contain the network scenario types to be executed and the corresponding test parameters. The communication pair list generation module is used to automatically generate a communication pair list required for testing based on the node information and the network scenario type. The communication pair list is used to map the one-to-one correspondence between multiple client nodes and server nodes. The connection management module is used to establish and manage connections with all nodes based on the node information in the communication pair list; An automated test execution module is used to automatically execute performance test commands sequentially on the corresponding client node and server node based on the communication pair list, test parameters and the established connection, so as to generate network traffic and obtain raw test output; The data acquisition and analysis module is used to automatically extract performance indicators from the original test output, analyze the performance indicators, and generate a structured test report.

10. A chip comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

11. A network interface card, characterized in that, It includes the chip and interface as described in claim 10, wherein the chip communicates externally through the interface.

12. A computing device, characterized in that, It includes a processor, a chip as described in claim 10, or a network interface card as described in claim 11, wherein the chip or the network interface card is used to schedule packets to the processor or the chip or the network interface card itself for processing, and the processor is used to process the packets scheduled by the chip or the network interface card.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that are loaded and executed by a processor to perform the operations described in any one of claims 1-8.

Images