Firewall multi-network structure automatic test system and method based on AI generated script

Abstract

The application discloses a kind of based on AI generation script's firewall multi-network structure automation test system and method, a kind of based on AI generation script's firewall multi-network structure automation test system, including work host, management network switch, first firewall to be measured, second firewall to be measured and five auxiliary test equipment, the work host, the management network switch, the first firewall to be measured, the second firewall to be measured and the five auxiliary test equipment are all connected by wired network, and standardization reusable test networking is built.The application is realized by AI technology the full automatic generation of test script and the autonomous perfection of test case, only needs to input test case document and online running firewall equipment, without manual coding and preset script template, can automatically parse case, collect firewall actual configuration information, end-to-end generation directly executable test script, while can automatically traverse all functions of firewall Web management page.

Description

Technical Field

[0001] This invention relates to the field of railway signal security technology, specifically to an automated testing system and method for firewall multi-network structures based on AI-generated scripts. Background Technology

[0002] Firewalls, as core protective devices for network security boundaries, are widely deployed in network architectures across various industries. In railway signaling systems, firewalls are critical devices for ensuring the secure operation of the network. In actual engineering deployments, four common network topologies are used: Access switching, virtual bridging, link aggregation, and routing. All railway signaling subsystems covered by these firewalls require comprehensive verification of the effectiveness of functions such as interface forwarding, VLAN isolation, routing forwarding, security policies, and load balancing to ensure the secure and stable operation of the railway signaling system. Currently, firewall testing in the industry generally adopts a combination of manual testing and semi-automated scripting. This involves manually building the test topology, writing test scripts, configuring network modes, and analyzing test results. With the application of artificial intelligence technology in testing, some AI-assisted testing solutions have emerged, but these can only achieve simple natural language to code conversion and still rely on fixed templates and manual parameter configuration. Existing firewall-related technology patents are mostly concentrated on AI attack detection, defense strategy optimization, firewall hardware architecture, and virtualization security, with relatively few solutions specifically for automated networking testing of firewalls with multiple network structures.

[0003] Existing firewall testing solutions have several shortcomings in practical applications. Test scripts rely entirely on manual writing, requiring pre-set script templates, configuration parameters, and code frameworks. They cannot be automatically generated solely from test case documents and online firewall devices, resulting in long testing cycles, high technical skill requirements for testers, and poor compatibility with different firewall models. Test cases are all manually written and cannot be automatically read from the firewall's web management page to retrieve actual functions, interface types, network modes, and policy options, leading to numerous testing blind spots and difficulty in accommodating the functional differences between firewalls from different vendors. Automated testing only supports single-network-structure testing, unable to automatically configure multiple network structures, let alone simultaneously configure and parallelize multiple network structures. It requires manual switching of network modes, resulting in low testing efficiency, a high risk of configuration errors, and difficulty in simulating real-world multi-service coexistence network scenarios. Link aggregation mode testing lacks a standardized automated process, requiring manual traffic injection and interface traffic monitoring. It cannot automatically initiate traffic, statistically analyze multi-interface traffic, or determine load balancing status, leading to highly subjective and inaccurate test results. Existing test environments often employ a single firewall architecture with a few auxiliary devices, lacking standardized fixed network topologies. The test scenarios differ significantly from actual engineering deployments, limiting the reference value of test results. Furthermore, existing solutions fail to establish a complete closed-loop system encompassing AI test case parsing, automatic script generation, automatic test case improvement, and integrated testing of multiple network structures, remaining at a single-point automation level. Web automation testing largely relies on third-party frameworks, resulting in limitations in test stability and adaptability, preventing the development of purely self-developed automated testing solutions. Moreover, the testing solutions do not integrate with the firewall's built-in debugging, diagnostic, and packet capture functions, failing to verify the effectiveness of packet capture while testing network structure functionality, leading to insufficient test coverage.

[0004] Therefore, there is an urgent need to develop an AI-driven automated testing solution that features standardized test networking, supports integrated automatic testing of multiple network structures, adopts a purely autonomous web automation implementation method, and can be linked with firewall packet capture functions. This solution aims to meet the high-efficiency, full-coverage, and unattended requirements of firewall mass production testing, version regression testing, and multi-model compatibility testing. It is especially suitable for the high-standard network security testing requirements of railway signaling systems, providing reliable testing support for the network security of railway signaling systems. Summary of the Invention

[0005] The purpose of this invention is to provide an automated testing system and method for firewalls with multiple network structures based on AI-generated scripts, so as to solve the problems existing in the prior art mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an automated testing system for firewall multi-network structures based on AI-generated scripts, comprising a working host, a management network switch, a first firewall under test, a second firewall under test, and five auxiliary testing devices. The working host, the management network switch, the first firewall under test, the second firewall under test, and the five auxiliary testing devices are all connected via a wired network to construct a standardized and reusable test network.

[0007] The host computer is equipped with an AI testing platform, which is suitable for storing test case documents, running AI models, generating and executing automated test scripts, and summarizing test logs and reports. The host computer is also suitable for accessing the firewall's web management page via a browser's native remote debugging protocol. The host computer has a built-in pcap file parsing component, which is suitable for parsing packet capture files downloaded by the firewall.

[0008] The management network switch connects the management interfaces of the working host, the first firewall under test, the second firewall under test, and the five auxiliary test devices to form an independent management network segment. The independent management network segment is used for remote device login, command issuance, and data interaction.

[0009] The management interface of the first firewall under test is connected to the management network switch. The four service ports of the first firewall under test are respectively connected to the service network ports of the first auxiliary test device to the fourth auxiliary test device. The two optical fiber interfaces of the first firewall under test are directly connected to the two optical fiber interfaces of the second firewall under test. The direct connection link is used to build a link aggregation interconnection scenario.

[0010] The management interface of the second firewall under test is connected to the management network switch. The second firewall under test is used for remote configuration and status query. The fiber optic interface of the second firewall under test is connected to the corresponding interface of the first firewall under test. The second firewall under test participates in the link aggregation mode test.

[0011] All five auxiliary testing devices adopt a dual-NIC architecture. The first NIC of each auxiliary testing device is uniformly assigned to the management network segment and connected to the management network switch. The first NIC accepts remote login and command scheduling from the AI ​​testing platform. The second NIC of each auxiliary testing device serves as a service port directly connected to the service port of the corresponding firewall. The second NIC is used to build a service server, initiate network probing, and handle concurrent traffic.

[0012] Preferably, the AI ​​testing platform adopts a five-layer architecture, which consists of a test case access layer, an NLP semantic parsing layer, a firewall information collection layer, an intelligent script generation layer, and a script verification and optimization layer.

[0013] The test case access layer is adapted to receive raw unstructured test case documents in various formats. The test case access layer completes document reading, format parsing, and preliminary preprocessing.

[0014] The NLP semantic parsing layer consists of a large model and a self-developed NLP algorithm. The NLP semantic parsing layer is suitable for performing word segmentation, entity extraction, intent recognition, and decomposing test scenarios, test objects, network patterns, test steps and expected results on the original test cases, and converting unstructured text into standardized structured test case data.

[0015] The firewall information collection layer is adapted to automatically log in to the firewall web management page via Python and the browser's native remote debugging protocol, traverse the page's function menu, collect the actual interface parameters of the device, supported network modes, security policy templates, and packet capture function parameters to form a device capability feature library. At the same time, the firewall information collection layer logs in to the auxiliary test device via the SSH protocol to collect device configuration information and complete the test environment status verification.

[0016] The intelligent script generation layer is suitable for calling code to generate large models, integrating the structured use case data with the device capability feature library, automatically matching the test process framework of four network structures, and generating executable Python test scripts;

[0017] The script verification and optimization layer is suitable for using the Python abstract syntax tree module to perform syntax compilation verification on the generated script, detect syntax errors, undefined variables and missing library dependencies, automatically adjust the instruction execution sequence, eliminate redundant operations, and achieve adaptive parameter correction.

[0018] Preferably, the firewall information collection layer is equipped with a Web automation control module, which is built based on the Python language. The Web automation control module is adapted to establish a communication connection with the browser's remote debugging port through the WebSocket protocol, and call the browser's official remote debugging protocol interface to complete page access, element identification and configuration operations of the firewall's Web management page.

[0019] Preferably, the AI ​​testing platform supports fully automated testing of four network structures: Access switching mode, virtual bridge switching mode, routing mode, and link aggregation switching mode. The AI ​​testing platform also supports single-mode serial testing and multi-mode parallel testing.

[0020] During testing in Access switching mode, virtual bridge switching mode, and routing mode, the AI ​​testing platform is suitable for calling the firewall's built-in debugging, diagnostic, and packet capture functions to automatically start and stop packet capture, download pcap format packet capture files, and parse packets, simultaneously verifying the effectiveness of the network structure functions and the firewall's packet capture functions.

[0021] Preferably, during the link aggregation switching mode test, the AI ​​test platform is suitable for configuring the link aggregation group and corresponding load sharing mode of the two firewalls under test, the AI ​​test platform is suitable for initiating concurrent test traffic and collecting real-time traffic statistics of each aggregation member interface, and the AI ​​test platform is suitable for performing difference calculation and normalization comparison on the traffic data of each interface through a traffic approximation comparison algorithm, and generating a load balancing effectiveness judgment result based on a preset deviation threshold.

[0022] An automated testing method for firewalls with multiple network structures based on AI-generated scripts, implemented using any of the aforementioned automated testing systems for firewalls with multiple network structures based on AI-generated scripts, includes the following steps:

[0023] S1) The AI ​​testing platform loads the original unstructured test case document, completes the structured parsing of the test cases through natural language processing, extracts the test type, the device under test, the physical interface, the network parameters, the test actions and the expected result fields, and generates standardized structured test case data;

[0024] S2) The AI ​​testing platform automatically logs into the Web management page of the firewall under test, collects the actual interface parameters of the device, supported network modes, security policy templates and packet capture function parameters, and forms a device capability feature library.

[0025] S3) The AI ​​testing platform integrates the structured test case data with the device capability feature library, matches the test process framework corresponding to the network structure, generates an executable Python test script, and completes script syntax verification and timing optimization.

[0026] S4) The AI ​​testing platform executes the test script to complete the interface configuration, security policy configuration, connectivity verification and traffic testing of the corresponding network structure, and synchronously links with the firewall's built-in packet capture function to complete packet capture and message parsing.

[0027] S5) The AI ​​testing platform summarizes all test data, generates a comprehensive test report, and automatically resets the configuration of the firewall under test and the auxiliary testing equipment after the test is completed, restoring the initial test environment.

[0028] Preferably, after the equipment capability acquisition is completed in step S2, the step further includes an automatic test case improvement step, specifically:

[0029] The AI ​​testing platform compares the collected list of actual firewall supported functions with the original structured test cases item by item, and identifies missing test items, incomplete test steps, and ambiguous expected results in the original test cases through text similarity matching.

[0030] The AI ​​testing platform automatically generates complete test items for the missing test items, automatically completes the operation process for the incomplete test steps, automatically supplements the quantitative judgment criteria for the ambiguous expected results, generates iteratively improved test cases, and archives them.

[0031] Preferably, in step S4, the AI ​​testing platform supports parallel testing of four network structures. The AI ​​testing platform uses a multi-threaded concurrent scheduling mechanism to generate independent test script processes for each network structure. Each process occupies a different physical interface and the auxiliary testing device. The resources of each process do not conflict with each other, and the configurations do not interfere with each other. Each process executes the test independently and synchronously uploads the test results to the AI ​​testing platform.

[0032] Preferably, the testing of the link aggregation switching mode specifically includes the following steps:

[0033] The AI ​​testing platform is configured with link aggregation groups for two firewalls under test. The corresponding fiber optic interfaces are added to the same aggregation group, and the aggregation protocol and load sharing mode are selected. The aggregation group status and negotiation results are verified.

[0034] The AI ​​testing platform controls the auxiliary testing equipment to initiate concurrent traffic and periodically collects real-time traffic statistics values ​​of aggregated member interfaces;

[0035] The AI ​​testing platform uses a traffic approximation comparison algorithm to perform difference calculations and normalization comparisons on the traffic data of each interface, and generates a load balancing effectiveness judgment result based on a preset deviation threshold.

[0036] Preferably, the specific execution process of resetting the configuration of the firewall under test and the auxiliary testing equipment in step S5 is as follows:

[0037] The AI ​​testing platform shuts down all packet capture tasks initiated during testing, deletes configured security policies and special interface configurations, disbands link aggregation groups, and restores the default configuration of firewall interfaces. At the same time, the AI ​​testing platform shuts down the business service processes and traffic initiation processes on the auxiliary testing device, completely restoring the initial test state and supporting continuous polling tests of multiple batches and models of firewalls.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1) This application achieves fully automated generation of test scripts and autonomous improvement of test cases through AI technology. Only test case documents and online firewall devices need to be input. No manual coding or preset script templates are required. It can automatically parse test cases, collect actual firewall configuration information, and generate directly executable test scripts end-to-end. At the same time, it can automatically traverse all functions of the firewall's web management page, intelligently compare with the original test cases, supplement missing test items, improve test steps and expected judgments, effectively eliminate test blind spots, significantly reduce the technical threshold for testers, shorten the test preparation cycle, and adapt to the testing needs of firewalls from different manufacturers and models. In addition, this application completes the automated operation of the firewall web page by calling the browser's official remote debugging protocol, without relying on third-party web automation frameworks, resulting in stronger test stability and device adaptability.

[0040] 2) This application supports fully automated testing of four network structures: Access switching, virtual bridge, link aggregation, and routing. It supports both individual testing of a single network structure and parallel testing of multiple network structures simultaneously. No manual intervention is required for network switching. It can realistically simulate complex network scenarios with multiple services coexisting, greatly improving testing efficiency. Furthermore, this application adopts a standardized hierarchical network topology with dual firewalls and multiple auxiliary devices, which fits the actual deployment scenario of the project. For the link aggregation mode, it can automatically initiate concurrent traffic, collect interface traffic statistics, and determine the effectiveness of load balancing. The test results are objective and accurate, effectively avoiding errors caused by manual operation and improving the reliability of the test results.

[0041] 3) This application can be linked with the firewall's built-in debugging, diagnostic, and packet capture functions. During network structure and function testing, packet capture can be started and stopped simultaneously, packet file downloads can be completed, and packet parsing can be performed concurrently. This verifies the effectiveness of the firewall's own packet capture function while verifying network functions, resulting in more comprehensive test coverage and enabling a thorough investigation of firewall security vulnerabilities. In addition, this application has the ability to automatically roll back the test environment. After the test, all test configurations can be automatically cleared and the device can be restored to its initial state. It supports polling automated testing of multiple batches and models of firewalls, enabling unattended mass production testing. The whole system forms a complete closed-loop testing system, effectively improving the standardization and automation level of firewall testing, and providing solid testing support for scenarios with high network security requirements, such as railway signaling systems. Attached Figure Description

[0042] Figure 1 This is a diagram of the test system structure for this application;

[0043] Figure 2 This is a flowchart of the testing method for this application. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of the invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In the description of the invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] In the description of the invention, it should be noted that the execution order of the steps is not limited by the sequence number. The possible changes in the order of some steps, the synchronous execution of steps, and the split execution of steps are all within the scope of protection of this application.

[0048] Please see Figure 1-2 The present invention provides a technical solution: an automated testing system for firewall multi-network structures based on AI-generated scripts, characterized in that it includes a working host, a management network switch, a first firewall under test, a second firewall under test, and five auxiliary testing devices. The working host, management network switch, first firewall under test, second firewall under test, and five auxiliary testing devices are all connected through a wired network to construct a standardized and reusable test network.

[0049] The work host is equipped with an AI testing platform, which is suitable for storing test case documents, running AI models, generating and executing automated test scripts, and summarizing test logs and reports. The work host is also suitable for accessing the firewall's web management page via the browser's native remote debugging protocol. The work host has a built-in pcap file parsing component, which is suitable for parsing packet capture files downloaded by the firewall.

[0050] The management network switch connects the working host, the management interface of the first firewall under test, the management interface of the second firewall under test, and the management interfaces of five auxiliary test devices to form an independent management network segment. The independent management network segment is used for remote device login, command issuance and data interaction.

[0051] The management interface of the first firewall under test is connected to the management network switch. The four service ports of the first firewall under test are respectively connected to the service network ports of the first auxiliary test device to the fourth auxiliary test device. The two fiber optic interfaces of the first firewall under test are directly connected to the two fiber optic interfaces of the second firewall under test. The direct connection link is used to build a link aggregation interconnection scenario.

[0052] The management interface of the second firewall under test is connected to the management network switch. The second firewall under test is used for remote configuration and status query. The fiber optic interface of the second firewall under test is connected to the corresponding interface of the first firewall under test. The second firewall under test participates in the link aggregation mode test.

[0053] All five auxiliary testing devices adopt a dual-NIC architecture. The first NIC of each auxiliary testing device is uniformly assigned to the management network segment and connected to the management network switch. The first NIC accepts remote login and command scheduling from the AI ​​testing platform. The second NIC of each auxiliary testing device serves as a service port directly connected to the corresponding firewall's service port. The second NIC is used to build a service server, initiate network probing, and handle concurrent traffic.

[0054] Specifically, the standardized fixed network architecture of this application, consisting of dual firewalls and multiple auxiliary testing devices, uses a unified management network switch to construct an independent management network segment, achieving physical separation between the management channel and the service channel. This ensures the stability of the test control link while enabling independent transmission of service test traffic. It can realistically simulate the layered network deployment of an engineering site, avoiding the problem of excessive deviation between traditional single-device simple networking and actual scenarios, and improving the engineering reference value of test results. The direct connection architecture of the dual firewalls can directly support the testing of high-availability networking scenarios such as link aggregation. The dual network card auxiliary devices can simultaneously carry out dual functions of management scheduling and service traffic, without the need for additional test terminals, effectively simplifying the test environment setup process and reducing the cost of test hardware investment.

[0055] Specifically, the GE1 interface of the second firewall under test is connected to the management network switch for remote configuration and status query. The GE7 and GE8 fiber optic interfaces of the second firewall under test are directly connected to the corresponding fiber optic interfaces of the first firewall under test, participating in link aggregation mode testing and constructing a high-availability test scenario for dual firewall interconnection. The second firewall under test is also configured with a gigabit Ethernet service interface, which is directly connected to the LAN2 service port of the fifth auxiliary test device. This service interface has the same function as the service Ethernet port of the first firewall under test, supporting configuration for different network operating modes and carrying service test traffic. The four service Ethernet ports of the first firewall under test and one service Ethernet port of the second firewall under test together constitute five independent service access channels, corresponding to the five auxiliary test devices. Multiple network structure test scenarios can be deployed simultaneously on the two firewalls under test, providing independent physical interface resources for parallel testing of four network structures. This ensures that the configuration and traffic of each test process do not interfere with each other, fully supporting the networking requirements for simultaneous configuration and parallel automated testing of multiple network structures.

[0056] The AI ​​testing platform adopts a five-layer architecture, which consists of the test case access layer, the NLP semantic parsing layer, the firewall information collection layer, the intelligent script generation layer, and the script verification and optimization layer.

[0057] The test case access layer is suitable for receiving raw unstructured test case documents in various formats. The test case access layer completes document reading, format parsing and preliminary preprocessing.

[0058] The NLP semantic parsing layer consists of a large model and a self-developed NLP algorithm. The NLP semantic parsing layer is suitable for word segmentation, entity extraction, intent recognition, and decomposition of test scenarios, test objects, network patterns, test steps and expected results in the original test cases, and converts unstructured text into standardized structured test case data.

[0059] The firewall information collection layer is suitable for automatically logging into the firewall web management page via Python and the browser's native remote debugging protocol, traversing the page's function menu, collecting the actual interface parameters of the device, supported network modes, security policy templates, and packet capture function parameters to form a device capability feature library. At the same time, the firewall information collection layer logs into the auxiliary test device via the SSH protocol to collect device configuration information and complete the test environment status verification.

[0060] The intelligent script generation layer is suitable for calling code to generate large models, integrating structured test case data and device capability feature library, automatically matching test process frameworks of four network structures, and generating executable Python test scripts;

[0061] The script validation and optimization layer is suitable for using the Python Abstract Syntax Tree module to perform syntax compilation and validation on the generated scripts, detect syntax errors, undefined variables and missing library dependencies, automatically adjust the instruction execution sequence, eliminate redundant operations, and achieve adaptive parameter correction.

[0062] Specifically, this application adopts a five-layered AI testing platform architecture. Each layer has clear functional boundaries, performs its own duties, and links with each other. From test case access, semantic parsing, and device information collection to script generation and verification optimization, a complete automated link is formed. It can realize end-to-end automatic conversion from unstructured test cases to executable scripts without manual intervention in intermediate processing. The layered architecture has good scalability and maintainability. Each module can be independently iterated and optimized. The device information collection layer can directly connect to firewall devices of different manufacturers and models, and the script verification optimization layer can adaptively correct the configuration logic differences of different devices, effectively improving the platform's adaptability to multiple types of devices.

[0063] The firewall information collection layer is equipped with a Web automation control module. The Web automation control module is built based on the Python language and is suitable for establishing a communication connection with the browser's remote debugging port via the WebSocket protocol. It calls the browser's official remote debugging protocol interface to complete page access, element identification, and configuration operations on the firewall's Web management page.

[0064] Specifically, this application's web automation control module, built on the Python language, directly calls the browser's official remote debugging protocol interface via the WebSocket protocol to complete page operations. This eliminates the dependence on third-party web automation frameworks, avoiding test stability issues caused by version iterations and insufficient compatibility of third-party frameworks. At the same time, it reduces external dependent components and lowers the deployment complexity of the test environment. The native protocol call method provides higher accuracy in identifying elements on the firewall's web management page, stronger stability in executing configuration operations, and better adapts to the customized management interfaces of various dedicated firewalls, ensuring the continuous and reliable operation of automated testing.

[0065] The AI ​​testing platform supports fully automated testing of four network structures: Access switching mode, virtual bridge switching mode, routing mode, and link aggregation switching mode. The AI ​​testing platform supports single-mode serial testing and multi-mode parallel testing.

[0066] During testing in Access switching mode, virtual bridge switching mode, and routing mode, the AI ​​testing platform is suitable for calling the firewall's built-in debugging, diagnostic, and packet capture functions to automatically start and stop packet capture, download pcap format packet capture files, and parse packets, simultaneously verifying the effectiveness of the network structure functions and the firewall's packet capture functions.

[0067] Specifically, this application supports fully automated testing of four mainstream network structures, while being compatible with both single-mode serial and multi-mode parallel testing modes. The testing method can be flexibly selected according to actual testing needs, greatly improving the flexibility of test scheduling. During the testing of Layer 2 switching and Layer 3 routing modes, the built-in packet capture function of the firewall is synchronously linked, eliminating the need to deploy additional bypass packet capture devices. This allows for the simultaneous completion of network forwarding function verification and device-specific packet capture function verification. A single test can cover two types of verification targets, expanding the test coverage dimension and avoiding the efficiency loss caused by repeatedly building test scenarios.

[0068] During the link aggregation switching mode test, the AI ​​test platform is suitable for configuring the link aggregation group and corresponding load sharing mode of two firewalls under test. The AI ​​test platform is suitable for initiating concurrent test traffic and collecting real-time traffic statistics of each aggregation member interface. The AI ​​test platform is suitable for performing difference calculation and normalization comparison on the traffic data of each interface through the traffic approximation comparison algorithm, and generating the load balancing effectiveness judgment result based on the preset deviation threshold.

[0069] Specifically, this application implements fully automated testing for link aggregation scenarios. It can automatically complete the aggregation group configuration and protocol negotiation verification of dual-end firewalls without requiring manual login to each device for configuration, thus reducing the probability of manual configuration errors. By automatically collecting real-time traffic data from aggregation member interfaces and combining it with a traffic similarity comparison algorithm, it completes the quantitative judgment of load balancing effectiveness, replacing the traditional method of manually checking traffic and subjectively judging the balancing effect. This eliminates subjective errors in human judgment and improves the accuracy and objectivity of link aggregation test results.

[0070] According to another aspect of this application, an automated testing method for firewalls with multiple network structures based on AI-generated scripts is also provided. This method, implemented based on the aforementioned automated testing system for firewalls with multiple network structures based on AI-generated scripts, includes the following steps:

[0071] S1) The AI ​​testing platform loads the original unstructured test case documents, completes the structured parsing of the test cases through natural language processing, extracts the test type, the device under test, the physical interface, the network parameters, the test actions, and the expected result fields, and generates standardized structured test case data.

[0072] S2) The AI ​​testing platform automatically logs into the web management page of the firewall under test, collects the actual interface parameters of the device, supported network modes, security policy templates and packet capture function parameters, and forms a device capability feature library.

[0073] The S3 AI testing platform integrates structured test case data with a device capability feature library, matches the test process framework with the corresponding network structure, generates executable Python test scripts, and completes script syntax verification and timing optimization.

[0074] The S4 AI testing platform executes test scripts to complete interface configuration, security policy configuration, connectivity verification, and traffic testing for the corresponding network structure, while simultaneously linking with the firewall's built-in packet capture function to complete packet capture and message parsing.

[0075] The S5 AI testing platform aggregates all test data, generates a comprehensive test report, and automatically resets the configuration of the tested firewall and auxiliary test equipment after the test is completed, restoring the initial test environment.

[0076] Specifically, the fully automated testing method of this application covers the entire closed loop from test case parsing, device information collection, script generation to test execution and environment recovery. Only initial input of test case documents and online devices is required to complete the entire testing process unattended, significantly reducing manual operation links in the testing process and effectively shortening the overall testing cycle. The standardized step-by-step execution process has clear logic and reasonable timing, which can ensure the standardization and reproducibility of the testing process, and facilitate batch reuse in scenarios such as production line testing and version regression testing, supporting large-scale testing needs.

[0077] After completing the equipment capability acquisition in step S2, the process also includes an automatic test case improvement step, which is as follows:

[0078] The AI ​​testing platform compares the collected list of actual firewall supported functions with the original structured test cases item by item, and identifies missing test items, incomplete test steps, and ambiguous expected results in the original test cases through text similarity matching.

[0079] The AI ​​testing platform automatically generates complete test entries for missing test items, automatically completes the operation process for incomplete test steps, automatically supplements quantitative judgment criteria for ambiguous expected results, generates iteratively improved test cases, and archives them.

[0080] Specifically, this application executes an automatic test case improvement process simultaneously after the equipment information is collected. By comparing the actual functions supported by the equipment with the original test cases one by one, it can accurately identify missing items, incomplete steps, and ambiguous criteria in the original test cases, and automatically complete the corresponding test content. This effectively eliminates the test blind spots caused by manually written test cases and improves the completeness and coverage of test cases. The automatically improved test cases can directly match the real functional characteristics of the device under test, avoiding the problem of mismatch between general test cases and the actual functions of the device, and ensuring the relevance and effectiveness of the test content.

[0081] In step S4, the AI ​​testing platform supports parallel testing of four network structures. The AI ​​testing platform uses a multi-threaded concurrent scheduling mechanism to generate independent test script processes for each network structure. Each process occupies different physical interfaces and auxiliary test equipment. The resources of each process do not conflict with each other, and the configurations do not interfere with each other. Each process executes the test independently and uploads the test results to the AI ​​testing platform synchronously.

[0082] Specifically, this application employs a multi-threaded concurrent scheduling mechanism to achieve parallel testing of various network structures. Each test process occupies an independent physical interface and auxiliary test equipment, ensuring that test resources do not conflict and configuration operations do not interfere with each other. This allows for simultaneous testing of multiple network scenarios on the same firewall under test, significantly improving the testing efficiency of a single device and reducing the overall test time. Each process executes independently and reports results independently, ensuring that an anomaly in a single test process will not affect the operation of other test processes, thus guaranteeing the overall stability of parallel testing and facilitating the classification, statistics, and problem localization of test results.

[0083] The testing of link aggregation switching mode specifically includes the following steps:

[0084] The AI ​​testing platform is configured with link aggregation groups for two firewalls under test. The corresponding fiber optic interfaces are added to the same aggregation group, and the aggregation protocol and load balancing mode are selected. The aggregation group status and negotiation results are verified.

[0085] The AI ​​testing platform controls auxiliary testing equipment to initiate concurrent traffic and periodically collects real-time traffic statistics from aggregated member interfaces;

[0086] The AI ​​testing platform uses a traffic approximation comparison algorithm to perform difference calculations and normalization comparisons on the traffic data of each interface, and generates a load balancing effectiveness judgment result based on a preset deviation threshold.

[0087] Specifically, the testing process for the link aggregation mode in this application is executed step by step in the order of configuration verification, traffic initiation, data collection, and result judgment. Each step is completed automatically by the platform, with smooth process connection and reasonable timing, which can fully cover the entire process of link aggregation function from deployment to verification. By verifying the aggregation group status and negotiation results before initiating test traffic, invalid tests caused by configuration failure can be avoided, improving the rigor of the testing process. At the same time, the quantitative deviation threshold judgment method can directly output clear test conclusions without the need for manual secondary analysis.

[0088] The specific execution process for resetting the configuration of the firewall under test and auxiliary test equipment in step S5 is as follows:

[0089] The AI ​​testing platform shuts down all packet capture tasks started during testing, deletes configured security policies and special interface configurations, disbands link aggregation groups, and restores the default configuration of firewall interfaces. At the same time, the AI ​​testing platform shuts down the business service processes and traffic initiation processes on the auxiliary testing devices, completely restoring the initial testing state and supporting continuous polling tests of multiple batches and models of firewalls.

[0090] Specifically, this application automatically performs a full configuration reset operation after testing, which can sequentially clean up packet capture tasks, security policies, interface special configurations and link aggregation groups generated during the test, restore the default configuration of firewall interfaces, and simultaneously shut down test services and traffic processes on auxiliary devices, completely restoring the initial state of the test environment. There is no need for manual configuration cleanup on each device, avoiding interference from residual configurations to subsequent tests. The automated environment reset capability can support continuous polling tests of multiple batches and multiple models of firewalls, providing a basic guarantee for unattended, mass production testing modes and improving the reuse efficiency of test resources.

[0091] Example 1

[0092] This embodiment provides a detailed description of the automated testing method for firewalls with multiple network structures based on AI-generated scripts and the corresponding testing system of the present invention. The complete technical implementation process is as follows.

[0093] I. Hardware Components and Connection Topology of the Automated Testing Environment

[0094] The test environment in this embodiment includes one Windows working host, one management network switch, two firewalls under test, and five auxiliary test devices. The Windows working host, management network switch, first firewall under test, second firewall under test, and five auxiliary test devices are all connected through a wired network to build a standardized and reusable test network.

[0095] The Windows workstation is equipped with an AI testing platform, which stores test case documents, runs AI models, generates and executes automated test scripts, and summarizes test logs and reports. The Windows workstation also has a Google Chrome browser installed, which can access the firewall's web management page via the official Google Chrome Remote Debugging Protocol (CDP). The Windows workstation also has a built-in pcap file parsing component, which is used to parse packet capture files downloaded by the firewall and can accurately process packet capture data from dedicated firewalls.

[0096] The management network switch connects the Windows work host, the GE1 management ports of the two firewalls under test, and the LAN1 management ports of the five auxiliary test devices to form an independent management network segment. In this embodiment, the management network segment uses 192.168.40.0 / 24 for remote device login, command issuance, and data interaction, simulating the management network segment architecture in actual engineering to ensure consistency between the test environment and the actual deployment.

[0097] The GE1 interface of the first firewall under test is connected to the management network switch to receive test commands and feedback test data. The GE3 to GE6 gigabit Ethernet ports of the first firewall under test are respectively connected to the LAN2 service network ports of auxiliary test devices 1 to 4 to carry out service tests of four network structures. The GE7 and GE8 of the first firewall under test are gigabit / megabit compatible fiber optic ports, which are directly connected to the GE7 and GE8 fiber optic ports of the second firewall under test to build a link aggregation interconnection scenario.

[0098] The GE1 interface of the second firewall under test is connected to the management network switch for remote configuration and status query; the GE7 and GE8 optical fibers of the second firewall under test are connected to the corresponding interfaces of the first firewall under test to participate in the link aggregation mode test and realize the high availability scenario simulation of the interconnection of the two firewalls.

[0099] All five auxiliary testing devices adopt a dual-NIC architecture. The LAN1 NIC of each auxiliary testing device is uniformly assigned to the management network segment and connected to the management network switch to accept remote login and command scheduling from the AI ​​testing platform. The LAN2 NIC of each auxiliary testing device is directly connected to the corresponding firewall's service port as a service port to build TCP / UDP servers, initiate Ping probes, and initiate concurrent traffic to simulate real business access scenarios.

[0100] II. Implementation of AI-Powered Dynamic Generation of Automated Test Scripts

[0101] All test logic in this embodiment is developed using Python. All firewall web page operations, remote device interaction, script generation, and execution are completed using self-developed technical solutions. The core technology stack and components are as follows: Python is the primary development language, used to implement all logic control, script generation, and data processing; natural language processing uses the HuggingFace large model combined with self-developed NLP word segmentation, entity extraction, and intent recognition algorithms to parse test cases and extract key test parameters; remote device interaction uses the Paramiko library for SSH remote login to assist in testing devices and executing traffic deployment and service start / stop commands; web automation uses Python. Hon integrates with the official Google Chrome Remote Debugging Protocol (CDP) to automate access and operation of firewall web pages without the need for third-party frameworks; network connectivity testing uses the ping3 and subprocess libraries for automated Ping probes; packet capture file parsing uses the dpkt library to parse .pcap format packet capture files and extract packet information; script generation and syntax verification use the CodeLlama code generation model combined with AST abstract syntax tree to generate test scripts and perform syntax verification and timing optimization; data structured storage uses JSON and Pandas for test case parsing, parameter storage, and result archiving.

[0102] The AI ​​testing platform adopts a five-layer architecture, with each layer working together to achieve the goal of automatically generating executable test scripts without templates, relying solely on test cases and devices. The functions of each layer are as follows:

[0103] 1. Test Case Access Layer: Receives raw unstructured test case documents in Word, TXT, and Excel formats, and performs document reading, format parsing, and preliminary preprocessing to provide a foundation for subsequent semantic parsing.

[0104] 2. NLP Semantic Parsing Layer: Composed of the HuggingFace large model and self-developed NLP algorithms, it performs word segmentation, entity extraction, and intent recognition on the original test cases, automatically decomposes the test scenarios, test objects, network patterns, test steps, and expected results, and converts unstructured text into standardized structured test case data.

[0105] 3. Firewall Information Acquisition Layer: Using Python combined with the ChromeCDP protocol, the system automatically logs into the firewall's web management page, traverses the page's function menus, and collects the device's actual interface parameters, supported network modes, security policy templates, packet capture function parameters, etc., to form a device capability feature library. Simultaneously, it logs into the auxiliary test device via the SSH protocol to collect device configuration information and complete the test environment status verification.

[0106] 4. Intelligent Script Generation Layer: Calls CodeLlama code to generate a large model, integrates structured test case data and device capability feature library, automatically matches the test process framework of four network structures, and generates executable Python test scripts line by line, including the entire process of browser startup, web login, interface configuration, packet capture control, traffic testing, result judgment, and configuration rollback.

[0107] 5. Script Validation and Optimization Layer: The generated scripts are compiled and validated using the Python AST (Abstract Syntax Tree) module to detect syntax errors, undefined variables, missing library dependencies, and other issues; the execution order of instructions is automatically adjusted to eliminate redundant and duplicate login and invalid query commands; and adaptive parameter correction is implemented to adapt to the differences in web page elements and command logic of different firewall vendors, ensuring that the scripts can be run directly.

[0108] The complete steps for dynamically generating test scripts using AI are as follows:

[0109] Step 1: Structured parsing of raw unstructured test cases. The AI ​​testing platform loads local test case documents using a Python file reading function, calls a self-developed NLP entity extraction function to extract fixed key fields such as test type, device under test, physical interface, network parameters, test actions, and expected results from the test case text; after extraction, it is automatically converted into a standardized JSON structured test case dictionary and stored in memory variables for subsequent script generation and use.

[0110] Step 2: Python automatically logs into the firewall via the ChromeCDP protocol to collect the device's actual capabilities. The AI ​​testing platform encapsulates a self-developed multi-protocol login general function. First, it launches the Google Chrome browser via a system call and enables remote debugging mode. The launch command is `chrome.exe --remote-debugging-port=9222`, and the port can be customized. It establishes a connection with the Chrome remote debugging port via the WebSocket protocol and calls the CDP interface to automatically open the firewall's web management address. Then, it locates the DOM elements of the username, password, and login buttons, automatically filling in the username and password based on element ID, name, or XPath path and triggering the login operation. After successful login, it waits for the page to load and automatically traverses all function menus on the firewall's web management page, including interface management, VLAN configuration, routing configuration, link aggregation, security policies, and debugging diagnostics (packet capture). It calls a self-developed page element parsing function to extract key information from each functional module, including the working modes supported by each GE interface, supported link aggregation protocols, load balancing algorithm types, security policy matching dimensions, and interfaces and parameters supported by the packet capture function. Finally, it saves the captured firewall capability feature library as a structured dictionary and performs field association matching with the structured test cases from the first step to ensure that subsequent script generation adapts to the actual capabilities of the device.

[0111] Step 3: AI Large-Scale Model Scene Matching and Script Framework Generation. The CodeLlama code generation and inference interface is invoked, inputting two types of prompts: first, structured test case business process prompts, including test scenarios, steps, and parameters; second, prompts regarding the current firewall's actual interface capabilities and model characteristics, including supported network modes, interface parameters, and web page element information. The large-scale model automatically matches the standard test process framework corresponding to the network structure based on these two types of inputs, generating the overall script structure, specifically including environment initialization code, multi-device login session establishment code, business server deployment code, interface network mode configuration code, security policy distribution code, connectivity detection and packet capture download code, result judgment and log saving code, and configuration rollback and recovery code.

[0112] Step 4: Automatically populate configuration commands and business parameters. The AI ​​testing platform automatically calls the parameter mapping matching function to accurately match the parameters in the structured test cases with the parameters in the device capability feature library, automatically completing the script parameter population. It maps the interface numbers in the test cases to the corresponding business devices and automatically fills back parameters such as VLAN numbers, IP addresses, and TCP 3333 port into the corresponding positions in the script. At the same time, based on the differences in firewall vendor web page elements and configuration logic, it automatically adapts the execution order and element location method of configuration operation commands to ensure that the script can be adapted to firewalls from different vendors.

[0113] Step 5: Automatic Script Syntax Validation and Logic Optimization. The generated script is compiled and validated using the Python Abstract Syntax Tree (AST) module. Syntax validation functions are called to detect and automatically correct syntax errors, undefined variables, missing library dependencies, and other issues. The execution order of instructions is automatically adjusted to ensure logical flow, such as launching the Chrome browser before logging into the firewall, setting up the business server before configuring the interface mode, and starting packet capture before executing business tests. Redundant and repetitive login operations and invalid query commands are eliminated to optimize script execution efficiency. The final output is a complete automated test Python script file that can be run directly by double-clicking and requires no manual modification.

[0114] III. AI-Automatic Test Case Improvement Implementation Process

[0115] This section is developed entirely in Python and does not rely on any third-party web automation frameworks. Its core functionality relies on the ChromeCDP protocol and a self-developed text similarity matching algorithm to achieve automatic iterative improvement of test cases.

[0116] The AI-automated login process for the firewall management interface is as follows: The AI ​​testing platform pre-stores the encrypted firewall management IP, web port, username, and password, eliminating the need for manual input by the user; the platform calls a system call function to launch the Google Chrome browser and enable remote debugging mode on port 9222; after establishing a connection via the WebSocket protocol, it automatically opens the firewall's web management address; the CDP interface Runtime.evaluate() is called to locate the DOM nodes of the username input box, password input box, and login button, automatically filling in the encrypted username and password based on element ID, name, or XPath path without parsing the HTML source code, and triggering the login button click event; after login, the CDP interface Page.waitForLoadEvent() is called to wait for the page to load completely, and the DOM structure of the page navigation menu is obtained through the CDP interface, automatically locating and locking the first and second-level menus such as interface configuration, network mode, link aggregation, security policy, and debugging diagnosis (packet capture), preparing for subsequent function traversal and information collection.

[0117] The AI-driven process for automatically identifying network modes and functions in the management interface is as follows: After logging into the firewall's web management page, the complete DOM structure of the page is obtained through the CDP interface DOM.getDocument(). Functional nodes such as the left navigation bar, dropdown options on each configuration page, and interface mode selection dropdown boxes are automatically traversed. A self-developed text extraction function is called to extract text tags from each functional node through the CDP interface, filtering out keywords related to network modes and device functions, including Access mode, virtual bridge mode, bridging mode, routing mode, link aggregation, LACP protocol, load balancing strategy, VLAN configuration, security policy, and packet capture. The identified functional items are categorized and archived to form a standardized function list, specifically divided into Layer 2 switching functions, Layer 3 routing functions, link aggregation high availability functions, security policy control functions, and debugging and diagnostic functions. The categorized function list is saved as a structured table in JSON format, forming a list of the firewall's actual supported functions for subsequent comparison with the original test cases.

[0118] The intelligent comparison with the original test cases is achieved through a combination of structured decomposition and self-developed text similarity matching. The specific steps are as follows: First, the original manual test cases are decomposed into independent test items. Each test item includes a test name, test scenario, involved functions, test steps, and expected results, forming a structured list of original test cases. At the same time, the identified list of firewall's actual supported functions is decomposed into standard function items with the same format as the original test case test items. Each item includes a function name, corresponding test scenario, and achievable test actions. Then, the self-developed text similarity matching algorithm is called to calculate the similarity between the original test case test items and the device function items. A similarity threshold of 85% is set, and a comparison is performed item by item: If the similarity is greater than or equal to the threshold, it is determined that the original test case already has the test item, and the test steps and expected results match the device function, so the test item is retained unchanged; if the similarity is less than the threshold, it is determined that the firewall actually has the function, but it is missing in the original test case, and it is marked as a test item to be supplemented. At the same time, the existing test items in the existing test cases are checked to detect whether there are items with missing test steps, missing parameters, or ambiguous expected results, and these are marked as items to be improved.

[0119] Based on the comparison results, the AI ​​automatically performs test case improvement operations, generating iterative improved test cases. Specific actions include: automatically adding missing test case entries; for test items to be supplemented, combining the actual functions of the firewall, automatically generating complete test items including test name, test scenario, involved functions, detailed test steps, and standardized expected results; supplementing network modes, configuration functions, load balancing modes, and packet capture function test items supported by the current firewall but not covered by the original test cases; automatically completing incomplete test steps; for items to be improved, referring to the logic of similar test items and combining the firewall's web page operation process, automatically completing missing steps such as interface configuration order, VLAN division, routing configuration, policy matching conditions, and packet capture start / stop steps; automatically standardizing and improving expected results; referring to the firewall's functional characteristics, automatically completing expected results such as connectivity, interception logs, packet capture traffic, and load balancing judgment criteria, ensuring that expected results are quantifiable and can be automatically judged; after completion, automatically generating a V2 version of the improved test case document, retaining the original version and marking modification records, covering the original test case library, realizing self-iterative improvement of test cases, and saving the improved test cases locally for subsequent script generation and use.

[0120] IV. Detailed Automated Testing Process for Four Network Structures

[0121] The automated testing of four network structures—Access switching mode, virtual bridge switching mode, routing mode, and link aggregation switching mode—was all conducted using Python combined with the ChromeCDP protocol to automate web page operations. The first three modes relied on the firewall's built-in debugging and diagnostic packet capture function to complete packet collection and analysis, and verified the effectiveness of the packet capture function. The link aggregation mode combined with the traffic statistics page to complete load balancing determination, with no manual intervention required throughout the entire process.

[0122] 1. Access Exchange Mode Automated Testing Process

[0123] This process is scheduled and executed by an AI-generated Python automated script. It calls functions such as Paramiko remote login, ping3 connectivity detection, ChromeCDPWeb automation, and dpktpcap parsing at the underlying level, and executes them automatically in a fixed sequence. At the same time, it also tests the built-in packet capture function of the firewall.

[0124] Step 1: Deployment of the business server. The script automatically logs into the auxiliary test device 2 via the ParamikoSSH protocol, calls the system port listening function, and automatically sets up the TCP server business listening port 3333 and the UDP server business listening port 3334 in the background. The script automatically saves the server process PID to ensure that the service is always listening during the test without manual intervention.

[0125] Step 2: Firewall Interface Configuration. The script launches the Chrome browser via Python + ChromeCDP protocol, automatically logs into the web management page of the first firewall under test, calls the CDP interface to locate the Network Configuration - Interface menu, and enters the GE3 and GE4 interface configuration pages; it automatically issues configuration operation commands to uniformly configure the GE3 and GE4 physical interfaces to Access switching mode and forcibly assigns the two interfaces to VLAN1; after the configuration operation is completed, it automatically clicks the save button, calls the CDP interface verification page to receive the configuration effect prompt, and confirms that the interface configuration is successful.

[0126] Step 3: Enable packet capture via the firewall's built-in interface. The script automatically accesses the firewall's system configuration - diagnostic tools - packet capture tool function page via the CDP interface, calls the CDP interface to locate the GE3 and GE4 interface checkboxes, selects the service interface to be tested, configures the packet capture direction to bidirectional uplink and downlink, and simultaneously collects inbound and outbound traffic. Click "Start Packet Capture" in the interface operation column to start the packet capture task; the packet capture task runs in the background, continuously collecting real traffic packets from the interface, while verifying that the firewall's packet capture function can start normally.

[0127] Step 4: Verify the effectiveness of isolation without pre-configured security policies. The firewall has a default all-deny policy. The script calls the ping3 library to control auxiliary test device 1 to continuously launch ICMPPing probes to the service IP of auxiliary test device 2 for 10 seconds, 1 packet per second. After the test sequence is reached, the script automatically enters the packet capture page through the CDP interface, clicks the "Save Now" button for the corresponding interface in System Configuration - Diagnostic Tools, and automatically downloads the packet capture files for GE3 and GE4 in standard .pcap format. AI calls the dpkt library to parse the pcap file and analyzes it to find that the GE3 packet capture file contains ICMP probe traffic launched outward from auxiliary test device 1, while the GE4 packet capture file has no downlink forwarding traffic. At the same time, the script reads the firewall security interception log through the CDP interface, matches the source and destination IP, VLAN, and port 5-tuple records, and determines that the firewall is performing Layer 2 isolation and defaults to blocking unauthorized cross-device access. It also verifies that the firewall packet capture function can terminate normally, the pcap file can be downloaded normally, and the packets can be recorded completely.

[0128] Step 5: Configure Access Mode Security Policy. The script automatically accesses the firewall policy configuration - security policy configuration interface via the CDP interface, and automatically fills in the policy parameters according to the AI ​​parsing test cases: source address is the service IP of auxiliary test device 1, destination address is the service IP of auxiliary test device 2, VLAN is VLAN1, service protocol is TCP port 3333, UDP port 3334, and ICMP protocol; after the policy configuration is completed, click Enable and Save, and verify the policy's effectiveness.

[0129] Step 6: Policy-based connectivity and bidirectional traffic verification. Re-execute the connectivity test by pinging auxiliary test device 2 from auxiliary test device 1, along with TCP requests on port 3333 and UDP requests on port 3334, for 10 seconds. Simultaneously, restart packet capture on interfaces GE3 and GE4 via the CDP interface. After the test, terminate packet capture and automatically download the pcap file. Use the dpkt library to parse the pcap file, confirming that bidirectional round-trip traffic exists on both interfaces. Simultaneously, read the firewall traffic permission logs via the CDP interface, matching the five-tuple passage records to determine that the interface Access mode configuration, VLAN partitioning, and security policy forwarding logic are all effective, and the firewall's built-in packet capture function is working correctly.

[0130] Step 7: Automatic Environment Rollback. After the single-mode test is completed, the script automatically closes the packet capture task, deletes the configured security policy, and restores the default configuration of the GE3 and GE4 interfaces via the CDP interface; it also closes the TCP / UDP server process of auxiliary test device 2 via the ParamikoSSH protocol, completing the test environment reset without affecting subsequent network structure tests.

[0131] 2. Automated Testing Process for Virtual Bridge Switching Mode

[0132] This mode reuses the complete testing framework of the Access mode and calls the same underlying function library. Only the interface configuration logic is adjusted differently. In the network configuration - interface - corresponding interface configuration mode, Vbridge (virtual bridge) is selected. The AI ​​testing platform can automatically identify the configuration menu path and element position of the virtual bridge mode through the CDP interface according to the actual firmware capabilities of the firewall, and automatically adapt the configuration instructions. There is no need to manually modify the script to adapt to different manufacturers' devices. The remaining verification steps and packet capture linkage test logic are completely consistent with the Access exchange mode.

[0133] 3. Automated testing process for routing mode

[0134] This process is designed for a firewall's Layer 3 routing and forwarding scenario. It uses Python and the ChromeCDP protocol to automate the entire web operation, relying on the firewall's built-in debugging, diagnostic, and packet capture functions to collect Layer 3 interface packets, while simultaneously testing the routing and forwarding functions and the firewall's built-in packet capture capabilities.

[0135] Step 1: Deploy the business server. The script logs into auxiliary test device 4 via the ParamikoSSH protocol, automatically calls the server deployment function, and sets up a TCP / UDP server listening service on fixed ports 3333 / 3334.

[0136] Step 2: Firewall Layer 3 Interface Configuration. The script launches the Chrome browser via Python + ChromeCDP protocol, automatically logs into the web management page of the first firewall under test, locates the Network Configuration - Interface menu, and switches the GE5 and GE6 interfaces from the default switching mode to routing mode; according to the test case, the parameters are fixed, and the IP configuration information is filled in through the CDP interface: the IP of the GE5 interface is configured as 172.16.40.1, and the subnet mask is 255.255.255.0; the IP of the GE6 interface is configured as 172.16.41.1, and the subnet mask is 255.255.255.0; after the configuration is distributed, the interface status page is read through the CDP interface to verify that the interface protocol is UP and the routing entries are generated normally.

[0137] Step 3: Enable Layer 3 interface packet capture. The script automatically accesses the firewall system configuration - diagnostic tools - packet capture tool page via the CDP interface, selects the GE5 and GE6 service interfaces, configures bidirectional packet capture, and starts the packet capture task to verify that the Layer 3 interface packet capture function can start normally.

[0138] Step 4: Verify the effectiveness of isolation without a security policy. Control auxiliary test device 3 (GE5 interface connected) to access the service address of auxiliary test device 4 (GE6 interface connected) across network segments, and perform Ping / TCP / UDP probes. After the test is completed, stop packet capture through the CDP interface and download the corresponding pcap format packet capture files for GE5 and GE6. The AI ​​calls the dpkt library to parse the pcap packets, which shows that there is access initiation traffic on the GE5 interface and no cross-network segment forwarding traffic on the GE6 interface. At the same time, read the firewall's Layer 3 access interception log through the CDP interface, match the source and destination network segments and protocol records, verify the default security isolation capability of the routing mode, and verify that the Layer 3 interface packet capture start / stop, download, and packet recording functions are normal.

[0139] Step 5: Configure security policy in routing mode. The script automatically enters the policy configuration - security policy configuration interface through the CDP interface, configures the cross-network segment security allow policy in routing mode, matches the source and destination network segments 172.16.41.0 / 24, 172.16.40.0 / 24, TCP3333, UDP3334, and ICMP service ports, saves and enables the policy.

[0140] Step 6: Post-policy clearance verification. Restart packet capture on GE5 and GE6 interfaces, and perform connectivity tests on auxiliary test device 3 (Ping / TCP / UDP) and auxiliary test device 4 across network segments. After the test is completed, stop packet capture, download and parse the pcap file, and verify the integrity of the Layer 3 bidirectional round-trip traffic. Read the firewall clearance log through the CDP interface, match the cross-network segment passage records, and determine that the Layer 3 interface configuration, IP routing, security policy, and built-in packet capture function are all normal.

[0141] Step 7: Automatic Environment Rollback. The script automatically deletes the IP configurations of GE5 and GE6 interfaces, restores the interface Layer 2 mode, cleans up security policies, and closes packet capture tasks via the CDP interface; it also shuts down the auxiliary test device 4 server process via the ParamikoSSH protocol, restoring the test environment and supporting continuous testing in multiple network modes.

[0142] 4. Automated testing process for link aggregation switching mode

[0143] This process is specifically designed for testing LACP link aggregation and load balancing between two interconnected firewalls. It uses Python and ChromeCDP protocol to automate the configuration of the two firewalls via the web and relies on the firewall traffic statistics page to determine the load balancing, eliminating the need for detailed packet capture of individual packets.

[0144] Step 1: Configure Dual Firewall Link Aggregation. The script uses Python multithreading to launch two Chrome browser instances, which automatically log into the web management pages of the first and second firewalls under test via the CDP protocol. It navigates to the Network Configuration - Interface - Add - Aggregate Interface Configuration menu, adds aggregation port ch1, and then adds the corresponding GE7 and GE8 fiber optic interconnect interfaces of the two firewalls to the same link aggregation group, with the aggregation group ID unified as ch1. It automatically selects the LACP protocol as the aggregation protocol and configures the load balancing mode as LACP-IP + port mixed load balancing mode. After configuration, it reads the aggregation group status page via the CDP interface to verify that the aggregation group members are in normal status, LACP neighbor negotiation is successful, and the aggregation link protocol is up.

[0145] Step 2: Concurrent Traffic Initiation. The script logs into auxiliary test device 5 via the ParamikoSSH protocol, automatically calls the concurrent traffic generation function, and simultaneously launches a large-volume concurrent packet attack on four terminals from auxiliary test devices 1 to 4. The traffic type is a mixture of TCP and UDP traffic, lasting for 30 seconds to simulate a high-concurrency business scenario.

[0146] Step 3: Traffic Statistics Collection. The script periodically logs into the two firewalls via the CDP interface, automatically enters the homepage - interface information list page, calls the page element extraction function, and collects real-time traffic statistics values ​​from the four aggregated member interfaces GE7 and GE8.

[0147] Step 4: Determine the effectiveness of load balancing. The AI-built-in traffic approximation comparison algorithm performs difference calculations and normalization comparisons on the traffic data of the four interconnected interfaces, setting an allowable threshold of ±10% for traffic deviation. If the traffic values ​​of the four interfaces are within the preset allowable threshold, the load balancing effect is deemed satisfactory. If the traffic of a certain interface is significantly higher or lower than the threshold, the load balancing is deemed to have failed, the test is automatically marked as abnormal, and the abnormal interface traffic data is recorded.

[0148] Step 5: Automatic Environment Rollback. After the test, the script logs into the two firewalls via the CDP interface, automatically disbands the link aggregation group, restores the independent configuration of the GE7 and GE8 fiber optic interfaces, and clears the interface traffic statistics. It stops the concurrent traffic attack on auxiliary test device 5 via the ParamikoSSH protocol, completing the automatic environment reset without affecting subsequent network structure tests.

[0149] V. Simultaneous Configuration and Parallel Testing Logic of Multiple Network Structures

[0150] The AI ​​testing platform adopts a multi-threaded concurrent scheduling mechanism. Relying on the Python threading module, it generates independent test script processes for Access switching mode, virtual bridge switching mode, routing mode, and link aggregation switching mode. Each process executes independently and does not interfere with each other.

[0151] In terms of process isolation, the four test processes occupy different service physical interfaces of the firewall and different auxiliary test devices respectively. Access / virtual line bridge uses GE3 and GE4 interfaces and auxiliary test devices 1 and 2; routing uses GE5 and GE6 interfaces and auxiliary test devices 3 and 4; link aggregation uses GE7 and GE8 interfaces and auxiliary test devices 4 and 5. The resources do not conflict with each other and the configurations do not interfere with each other, which can simulate the real scenario of multiple network modes coexisting.

[0152] In terms of parallel execution, Access, virtual bridge, and routing modes all independently call the Python+ChromeCDP protocol, start independent Chrome browser instances, log in to the firewall web management page, and independently perform operations such as interface configuration, packet capture start / stop, pcap download and parsing, and log comparison. In link aggregation mode, dual firewall configuration, traffic initiation, and traffic statistics determination are executed synchronously, with all four processes proceeding simultaneously.

[0153] In terms of results aggregation, after each test process independently completes the result judgment, the test data is uploaded to the AI ​​testing platform. The platform automatically aggregates all test results, generates an integrated comprehensive test report, and marks whether each network structure test has passed and whether the firewall packet capture function is normal. This enables the simultaneous configuration and parallel fully automated testing of four network structures, greatly improving testing efficiency.

[0154] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automated testing system for firewalls with multiple network structures based on AI-generated scripts, characterized in that, It includes a working host, a management network switch, a first firewall under test, a second firewall under test, and five auxiliary testing devices. The working host, the management network switch, the first firewall under test, the second firewall under test, and the five auxiliary testing devices are all connected via a wired network. The host computer is equipped with an AI testing platform, which is suitable for storing test case documents, running AI models, generating and executing automated test scripts, and summarizing test logs and reports; the host computer is also suitable for accessing the firewall's web management page via a browser's native remote debugging protocol. The host computer has a built-in pcap file parsing component, which is suitable for parsing packet capture files downloaded by the firewall; The management network switch connects the management interfaces of the working host, the first firewall under test, the second firewall under test, and the five auxiliary test devices to form an independent management network segment. The independent management network segment is used for remote device login, command issuance, and data interaction. The management interface of the first firewall under test is connected to the management network switch. The four service ports of the first firewall under test are respectively connected to the service network ports of the first auxiliary test device to the fourth auxiliary test device. The two optical fiber interfaces of the first firewall under test are directly connected to the two optical fiber interfaces of the second firewall under test. The direct connection link is used to build a link aggregation interconnection scenario. The management interface of the second firewall under test is connected to the management network switch. The second firewall under test is used for remote configuration and status query. The fiber optic interface of the second firewall under test is connected to the corresponding interface of the first firewall under test. The second firewall under test participates in the link aggregation mode test. All five auxiliary testing devices adopt a dual-NIC architecture. The first NIC of each auxiliary testing device is uniformly assigned to the management network segment and connected to the management network switch. The first NIC accepts remote login and command scheduling from the AI ​​testing platform. The second NIC of each auxiliary testing device serves as a service port directly connected to the service port of the corresponding firewall. The second NIC is used to build a service server, initiate network probing, and handle concurrent traffic.

2. The automated testing system for firewall multi-network structures based on AI-generated scripts according to claim 1, characterized in that, The AI ​​testing platform adopts a five-layer architecture, which consists of the test case access layer, the NLP semantic parsing layer, the firewall information collection layer, the intelligent script generation layer, and the script verification and optimization layer. The test case access layer is adapted to receive raw unstructured test case documents in various formats. The test case access layer completes document reading, format parsing, and preliminary preprocessing. The NLP semantic parsing layer consists of a large model and a self-developed NLP algorithm. The NLP semantic parsing layer is suitable for performing word segmentation, entity extraction, intent recognition, and decomposing test scenarios, test objects, network patterns, test steps and expected results on the original test cases, and converting unstructured text into standardized structured test case data. The firewall information collection layer is adapted to automatically log in to the firewall web management page via Python and the browser's native remote debugging protocol, traverse the page's function menu, collect the actual interface parameters of the device, supported network modes, security policy templates, and packet capture function parameters to form a device capability feature library. At the same time, the firewall information collection layer logs in to the auxiliary test device via the SSH protocol to collect device configuration information and complete the test environment status verification. The intelligent script generation layer is suitable for calling code to generate large models, integrating the structured use case data with the device capability feature library, automatically matching the test process framework of four network structures, and generating executable Python test scripts; The script verification and optimization layer is suitable for using the Python abstract syntax tree module to perform syntax compilation verification on the generated script, detect syntax errors, undefined variables and missing library dependencies, automatically adjust the instruction execution sequence, eliminate redundant operations, and achieve adaptive parameter correction.

3. The automated testing system for firewall multi-network structures based on AI-generated scripts according to claim 2, characterized in that, The firewall information acquisition layer is equipped with a Web automation control module, which is built based on the Python language. The Web automation control module is suitable for establishing a communication connection with the browser's remote debugging port through the WebSocket protocol, and calling the browser's official remote debugging protocol interface to complete page access, element identification and configuration operations of the firewall's Web management page.

4. The automated testing system for firewall multi-network structures based on AI-generated scripts according to claim 1, characterized in that, The AI ​​testing platform supports fully automated testing of four network structures: Access switching mode, virtual bridge switching mode, routing mode, and link aggregation switching mode. The AI ​​testing platform also supports single-mode serial testing and multi-mode parallel testing. During testing in Access switching mode, virtual bridge switching mode, and routing mode, the AI ​​testing platform is suitable for calling the firewall's built-in debugging, diagnostic, and packet capture functions to automatically start and stop packet capture, download pcap format packet capture files, and parse packets, simultaneously verifying the effectiveness of the network structure functions and the firewall's packet capture functions.

5. The automated testing system for firewall multi-network structures based on AI-generated scripts according to claim 1, characterized in that, During the link aggregation switching mode test, the AI ​​test platform is suitable for configuring the link aggregation group and corresponding load sharing mode of the two firewalls under test. The AI ​​test platform is suitable for initiating concurrent test traffic and collecting real-time traffic statistics of each aggregation member interface. The AI ​​test platform is suitable for performing difference calculation and normalization comparison on the traffic data of each interface through the traffic approximation comparison algorithm, and generating a load balancing effectiveness judgment result based on the preset deviation threshold.

6. An automated testing method for firewalls with multiple network structures based on AI-generated scripts, characterized in that, The automated testing system for firewall multi-network structures based on AI-generated scripts, as described in any one of claims 1 to 5, includes the following steps: S1) The AI ​​testing platform loads the original unstructured test case document, completes the structured parsing of the test cases through natural language processing, extracts the test type, the device under test, the physical interface, the network parameters, the test actions and the expected result fields, and generates standardized structured test case data; S2) The AI ​​testing platform automatically logs into the Web management page of the firewall under test, collects the actual interface parameters of the device, supported network modes, security policy templates and packet capture function parameters, and forms a device capability feature library. S3) The AI ​​testing platform integrates the structured test case data with the device capability feature library, matches the test process framework corresponding to the network structure, generates an executable Python test script, and completes script syntax verification and timing optimization. S4) The AI ​​testing platform executes the test script to complete the interface configuration, security policy configuration, connectivity verification and traffic testing of the corresponding network structure, and synchronously links with the firewall's built-in packet capture function to complete packet capture and message parsing. S5) The AI ​​testing platform summarizes all test data, generates a comprehensive test report, and automatically resets the configuration of the firewall under test and the auxiliary testing equipment after the test is completed, restoring the initial test environment.

7. The automated testing method for firewalls with multiple network structures based on AI-generated scripts according to claim 6, characterized in that, After the equipment capability acquisition is completed in step S2, the step of automatically improving test cases is also included, specifically: The AI ​​testing platform compares the collected list of actual firewall supported functions with the original structured test cases item by item, and identifies missing test items, incomplete test steps, and ambiguous expected results in the original test cases through text similarity matching. The AI ​​testing platform automatically generates complete test items for the missing test items, automatically completes the operation process for the incomplete test steps, automatically supplements the quantitative judgment criteria for the ambiguous expected results, generates iteratively improved test cases, and archives them.

8. The automated testing method for firewalls with multiple network structures based on AI-generated scripts according to claim 6, characterized in that, In step S4, the AI ​​testing platform supports parallel testing of four network structures. The AI ​​testing platform uses a multi-threaded concurrent scheduling mechanism to generate independent test script processes for each network structure. Each process occupies a different physical interface and the auxiliary testing equipment. Each process executes the test independently and synchronously uploads the test results to the AI ​​testing platform.

9. The automated testing method for firewalls with multiple network structures based on AI-generated scripts according to claim 8, characterized in that, The testing of link aggregation switching mode specifically includes the following steps: The AI ​​testing platform is configured with link aggregation groups for two firewalls under test. The corresponding fiber optic interfaces are added to the same aggregation group, and the aggregation protocol and load sharing mode are selected. The aggregation group status and negotiation results are verified. The AI ​​testing platform controls the auxiliary testing equipment to initiate concurrent traffic and periodically collects real-time traffic statistics values ​​of aggregated member interfaces; The AI ​​testing platform uses a traffic approximation comparison algorithm to perform difference calculations and normalization comparisons on the traffic data of each interface, and generates a load balancing effectiveness judgment result based on a preset deviation threshold.

10. The automated testing method for firewalls with multiple network structures based on AI-generated scripts according to claim 6, characterized in that, The specific execution process of resetting the configuration of the firewall under test and the auxiliary test equipment in step S5 is as follows: The AI ​​testing platform shuts down all packet capture tasks initiated during testing, deletes configured security policies and special interface configurations, disbands link aggregation groups, and restores the default configuration of firewall interfaces. At the same time, the AI ​​testing platform shuts down the business service processes and traffic initiation processes on the auxiliary testing device, completely restoring the initial test state and supporting continuous polling tests of multiple batches and models of firewalls.

Images