Method, device, equipment and medium for automatic testing of intrusion detection and prevention system

Abstract

The application discloses an automatic testing method, device, equipment and medium of an intrusion detection and prevention system, relates to the field of computers, and comprises the following steps: obtaining and analyzing a preset configuration file to obtain testing configuration parameters, testing case groups and a testing case list, and establishing a target testing environment according to the testing configuration parameters; performing initialization operation on each testing case group based on a preset sequence, and dynamically loading a testing script of a current testing case in the testing case list; determining an attack execution mode according to a testing type, initiating an attack test on a terminal under test through a preset mode according to the attack execution mode; obtaining a testing result from the terminal under test, determining a new current testing case based on the preset sequence, continuing to perform the step of dynamically loading the testing script corresponding to the current testing case in the testing case list, and generating a testing report corresponding to the target testing task until the testing cases in the testing case group are executed. All testing items of the IDPS system can be automatically covered.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to automated testing methods, apparatus, equipment and media for intrusion detection and prevention systems. Background Technology

[0002] Based on a large IDPS (Intrusion Detection and Prevention System), it contains several sub-modules, including Host IDS (Intrusion Detection System), CAN_IDS, Network IDS, etc. Each sub-module contains different sub-items. For example, Host IDS includes monitoring of the file system, processes, and system resources. Different items have different testing methods. For instance, testing the file system includes creating, deleting, modifying, and querying files, while testing the Network IDS involves sending attack packets to the target through an external network channel.

[0003] In conclusion, how to automatically cover all test items of the IDPS system is a problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an automated testing method, apparatus, equipment, and medium for intrusion detection and prevention systems (IDPS), capable of automatically covering all test items of the IDPS system. The specific solution is as follows: In a first aspect, this application provides an automated testing method for an intrusion detection and prevention system, applied to the control host of the intrusion detection and prevention system. The testing system for the intrusion detection and prevention system comprises the control host, an attacking host, and a terminal under test. The control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method, and the attacking host is connected to the terminal under test via a network. The method includes: Obtain and parse the preset configuration file of the target test task to obtain test configuration parameters, test case groups and test case lists in each test case group, and establish the target test environment according to the test configuration parameters; The test case groups are initialized according to a preset order, and the test scripts corresponding to the current test case in the test case list are dynamically loaded. The attack execution method is determined according to the test type of the current test case, and the attack test corresponding to the current test case is initiated to the terminal under test according to the attack execution method through a preset method. The test results corresponding to the current test case are obtained from the terminal under test. After the current test case is executed, a new current test case is determined based on the preset order. The step of dynamically loading the test script corresponding to the current test case in the test case list is continued until the test cases in the test case group are executed and the corresponding test report of the target test task is generated. The step of determining the attack execution method based on the test type of the current test case, and initiating an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method, includes: When the attack execution method is a preset routine test, the terminal under test is directly tested through the first communication method. When the attack execution method is a bus attack test, an abnormal CAN message is sent directly to the terminal under test through the first communication method to perform a bus attack test on the terminal under test.

[0005] Optionally, the test configuration parameters include attack host connection parameters, tested terminal connection parameters, test case group configuration parameters, and test environment configuration parameters; Accordingly, establishing the target test environment based on the test configuration parameters includes: A first communication connection is established between the attacker and the host through the Secure Shell protocol, the user's login information, and the attacker's host connection parameters. A second communication connection is established between the terminal under test and the terminal under test based on the connection parameters of the terminal under test, as well as the USB interface and / or CAN bus. The attacking host and the tested terminal are connected by creating a network hotspot; The target test environment is built based on the first communication connection, the second communication connection, and the network hotspot.

[0006] Optionally, establishing a second communication connection with the terminal under test based on the connection parameters of the terminal under test, and the USB interface and / or CAN bus, includes: The CAN bus configuration parameters are determined from the test configuration parameters; wherein, the CAN bus configuration parameters include the baud rate of the CAN channel of the controller area network, whether the CAN FD parameter is enabled, and any one or more of the whitelist message identifier IDs; Based on the whitelist message identifier ID list, corresponding communication rules for the CAN database are generated to establish the second communication connection based on the communication rules.

[0007] Optionally, the initialization operation of each test case group based on a preset order, and the dynamic loading of the test script corresponding to the current test case in the test case list, includes: Perform group-level initialization operations on the test case group to establish the execution environment for the test case group; The test case group is traversed to determine the current test case based on a preset order; The test script is determined based on the script path corresponding to the current test case; The execution interface of the test script is invoked to load the current test case.

[0008] Optionally, the step of determining the attack execution method based on the test type of the current test case, and initiating an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method, includes: When the attack execution method is a preset routine test, the terminal under test is directly tested through the second communication connection established through the USB interface. When the attack execution method is a network attack test, the attack host is controlled to generate target attack data through the first communication connection and send the target attack data to the terminal under test to perform a network attack test on the terminal under test. When the attack execution method is a bus attack test, an abnormal CAN message is sent directly to the terminal under test through the second communication connection established by the CAN bus to perform a bus attack test on the terminal under test.

[0009] Optionally, obtaining the test result corresponding to the current test case from the terminal under test, and determining a new current test case based on the preset order after the current test case has been executed, includes: Log files are retrieved from the terminal under test using the Android Debug Bridge ADB command, and the log files are parsed to obtain the test results of the intrusion detection and prevention system in the attack test of the current test case. Compare the test results with the preset test expectations; If the test result is consistent with the preset test expectation result, it is determined that the attack test of the current test case has passed, and the step of dynamically loading the test script corresponding to the current test case in the test case list continues to be executed; If the test result is inconsistent with the preset test expectation result, it is determined that the attack test of the current test case has failed, the failure information of the current test case is recorded, the corresponding test result is generated based on the failure information, a new current test case is determined, and the step of dynamically loading the test script corresponding to the current test case in the test case list continues to be executed.

[0010] Optionally, generating the test report corresponding to the target test task includes: By using a pre-defined report monitoring process, the execution information and test judgment results of each test case in the test case group are collected. The execution information and test results are rendered into a preset hypertext markup language template and summarized into a test report for the target test task.

[0011] Secondly, this application provides an automated testing device for an intrusion detection and prevention system, applied to the control host of the intrusion detection and prevention system. The testing system comprises the control host, an attacking host, and a terminal under test. The control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method. The attacking host is connected to the terminal under test via a network. The device includes: The environment setup module is used to obtain and parse the preset configuration file of the target test task to get the test configuration parameters, test case groups and test case lists in each test case group, and to set up the target test environment according to the test configuration parameters. The script loading module is used to perform initialization operations on each of the test case groups based on a preset order, and dynamically load the test script corresponding to the current test case in the test case list; The test initiation module is used to determine the attack execution method according to the test type of the current test case, and to initiate an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method. The report generation module is used to obtain the test results corresponding to the current test case from the terminal under test, determine a new current test case based on the preset order after the current test case is executed, and continue to execute the step of dynamically loading the test script corresponding to the current test case in the test case list until the test cases in the test case group are executed, and generate the test report corresponding to the target test task. Specifically, the test initiation module is used to directly test the terminal under test through the first communication method when the attack execution method is a preset normal test; and to directly send an abnormal CAN message to the terminal under test through the first communication method when the attack execution method is a bus attack test.

[0012] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the automated testing method for the intrusion detection and prevention system as described above.

[0013] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned automated testing method for an intrusion detection and prevention system.

[0014] In summary, this application obtains and parses the preset configuration file of the target test task to obtain test configuration parameters, test case groups, and test case lists within each test case group. It then establishes a target test environment based on the test configuration parameters. The application initializes each test case group according to a preset order and dynamically loads the test script corresponding to the current test case in the test case list. It determines the attack execution method based on the test type of the current test case and initiates an attack test corresponding to the current test case to the terminal under test using a preset method according to the attack execution method. It obtains the test result corresponding to the current test case from the terminal under test. After the current test case is executed, it determines a new current test case based on the preset order and continues executing the step of dynamically loading the test script corresponding to the current test case in the test case list until all test cases in the test case group are executed, generating a test report corresponding to the target test task. As can be seen, the control host of this application first obtains and parses the preset configuration file of the target test task, extracting test configuration parameters, test case groups, and detailed test case lists within each group. Based on these configuration parameters, it automatically establishes a collaborative test environment between the host, the attacking host, and the terminal under test. Subsequently, each test case group is initialized sequentially according to a preset order. For the current test case in each test case group, its corresponding test script is loaded using dynamic loading technology to execute the test task. During execution, based on the test type specified by the current test case, the corresponding attack execution method is determined, and an attack test corresponding to that test case is launched to the terminal under test through a preset attack path. After the attack is completed, the test results are obtained from the terminal under test, and the execution status of the current test case is determined. After the current test case is completed, the next new test case is determined in sequence, and the above dynamic loading and execution steps are repeated until all test cases in the test case group and even the entire test task have been executed. Finally, a complete test report is generated. In this way, through configuration-driven and dynamic script loading, the entire process of test task orchestration is automated, significantly reducing manual intervention. At the same time, by coordinating the control of the attack host and bus interface, unified scheduling of multiple types of attacks, such as network and bus attacks, is supported, effectively improving the efficiency and versatility of automated testing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1This is a flowchart of an automated testing method for an intrusion detection and prevention system disclosed in this application; Figure 2 This is a test system framework diagram of a specific intrusion detection and prevention system disclosed in this application; Figure 3 This is a schematic diagram of the structure of an automated testing device for an intrusion detection and prevention system disclosed in this application; Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Currently, large-scale IDPS systems contain sub-modules, including host IDS, CAN_IDS, and network IDS, etc. Each sub-module further distinguishes different sub-items. For example, host IDS includes monitoring of the file system, processes, and system resources. Different items have different testing methods. For instance, testing the file system includes adding, deleting, modifying, and querying files, while testing the network IDS involves sending attack packets to the target through an external network channel. To address the aforementioned technical problems, this application discloses an automated testing method, apparatus, device, and medium for intrusion detection and prevention systems, capable of automatically covering all test items of the IDPS system.

[0019] See Figure 1 As shown, this invention discloses an automated testing method for an intrusion detection and prevention system, applied to the control host of the intrusion detection and prevention system. The testing system comprises the control host, an attacking host, and a terminal under test. The control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method. The attacking host is connected to the terminal under test via a network. The method includes: Step S11: Obtain and parse the preset configuration file of the target test task to obtain test configuration parameters, test case groups and test case lists in each test case group, and establish the target test environment according to the test configuration parameters.

[0020] In this embodiment, an automated testing environment is built on the basis of the intrusion detection and prevention system. The directory structure of the automated testing system is shown below: ├── README.md ├── cases

Use Case Summary Directory

Template Directory

Tool Directory

[0021] Then, begin building such Figure 2 The target test environment shown requires establishing a first communication connection with the attacking host through the Secure Shell protocol, the user's login information, and the attacking host's connection parameters; establishing a second communication connection with the terminal under test based on the terminal under test's connection parameters, and a USB interface and / or CAN bus; connecting the attacking host and the terminal under test by constructing a network hotspot; and building the target test environment based on the first communication connection, the second communication connection, and the network hotspot.

[0022] Specifically, for the establishment of the first communication connection, the controlling host calls the SSHClient class in the paramiko library to create an SSH client instance, sets the missing host key policy to automatic acceptance, and then calls the connect method of the instance, passing in the IP address, port number, username and corresponding password of the attacking host as input parameters to establish a remote control session with the attacking host, enabling the controlling host to obtain the privilege to execute arbitrary shell commands on the attacking host.

[0023] For establishing the second communication connection, the control host executes the `adb devices` command through the `subprocess.run` interface under the Windows system. The returned results confirm that the terminal under test is successfully connected via USB cable and is online, thus establishing a debugging session based on the ADB protocol and opening a channel for sending shell commands and retrieving files to the terminal under test. If the project configuration file contains a `can_cfg` field, the CAN bus configuration parameters need to be determined from the test configuration parameters. These CAN bus configuration parameters include the baud rate of the CAN channel in the controller area network, whether CAN FD is enabled, and any one or more items from the whitelist message identifier ID list. Based on the whitelist message identifier ID list, corresponding communication rules for the CAN database are generated to establish the second communication connection. Specifically, the control host reads the baud rate and whitelist message identifier ID list, and by calling the initialization interface in the ControlCANFD dynamic link library under the `libs` directory, enables the CAN box device's channel and configures its parameters to establish the CAN bus connection path.

[0024] For the network connection between the attacking host and the tested terminal, the controlling host switches its wireless network card to hotspot mode by calling the operating system's network interface, creating a wireless LAN hotspot using the service set identifier and password specified in the configuration file. The controlling host then remotely sends commands to the attacking host to connect to this hotspot via an established SSH session, and simultaneously sends commands to the tested terminal via an ADB session to connect to the same hotspot, ensuring that the attacking host and the tested terminal are on the same LAN segment.

[0025] After completing the connection between the control host, the attacking host, and the terminal under test, a target test environment is set up with the control host as the center, controlling the attacking host via SSH connection, managing the terminal under test via USB and ADB connection, and having a direct network connection between the attacking host and the terminal under test.

[0026] Step S12: Initialize each test case group according to a preset order, and dynamically load the test script corresponding to the current test case in the test case list.

[0027] In this embodiment, a group-level initialization operation is performed on the test case group to establish the execution environment of the test case group; the test case group is traversed to determine the current test case based on a preset order; the test script is determined according to the script path corresponding to the current test case; and the execution interface of the test script is called to load the current test case.

[0028] Specifically, after setting up the target test environment, the control host immediately enters the test case execution phase, processing each test case group sequentially according to the order defined in the group configuration fields of the configuration file. For the current test case group to be processed, the control host first locates the bootstrap.py startup script in the test case group directory based on the directory path of the test case group. It then uses the spec_from_loader and module_from_spec functions in the importlib.util dynamic loading library to load the test script. Next, it calls the start interface function defined in the script, passing the parsed test configuration parameters and the private interface instance as parameters to establish the dedicated execution environment for the test case group. For example, based on the test case group configuration information, it issues instructions to the terminal under test through an ADB session to adjust the on or off status of each detection module of its intrusion detection and prevention system.

[0029] The start interface is defined as follows: def start(test_config, pri_api:BaseClass): """ Root-level global initialization: 1. Receive test_config passed from main. 2. Manage the cases / tmp directory: create it if it doesn't exist, otherwise clear its internal files. """ def stop(test_config, pri_api:BaseClass): """ Root-level global initialization: 1. Receive test_config passed from main. """ After group-level initialization, the control host iterates through the test case list corresponding to the test case group, determining the test cases to be executed one by one based on the index order in the list. For the determined current test case, the control host constructs the complete storage path of the test case script file in the cases directory based on the relative path recorded in the name field of the cases array. Subsequently, the control host uses the spec_from_file_location function in the importlib.util library to create a module loading specification based on the complete path, and then uses module_from_spec and the loader.exec_module method of the specification object to dynamically load the test case script into a callable module object in the current test process. After loading is complete, the control host obtains the defined test case initialization interface, test case execution interface, and test case cleanup interface in sequence, and calls them sequentially. The start test case initialization interface is called to complete the test case-level parameter preparation, the run test case execution interface is called to execute the core test logic of the test case in a blocking manner, and after the run interface completes execution and returns, the stop test case cleanup interface is called to complete the cleanup work. After the current test case completes execution, the control host selects the next test case from the list as the new current test case, repeating the dynamic loading and interface call process described above until all test cases in the test case group have been executed. The interface definition within the test case is as follows: def start(test_config, pri_api:BaseClass): """ Initialization function: Receives test_config passed from main. :param test_config: The test_config dictionary in the JSON file. """ def run(): pass def stop(): pass Step S13: Determine the attack execution method according to the test type of the current test case, and initiate an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method.

[0030] In this embodiment, after the control host loads and starts the current test case, it determines the corresponding attack execution method based on the directory path to which the test case script belongs and the preset test type identifier, and then initiates the test to the terminal under test through different attack paths.

[0031] In one specific embodiment, when the attack execution method is a preset routine test, the terminal under test is directly tested through the second communication connection established via the USB interface. Specifically, when the attack execution method of the current test case is determined to be a preset routine test, i.e., the test case is located in directories such as cases / baseline / general or cases / process monitoring / general, the test is mainly conducted through an ADB debugging session established via the USB interface. The control host directly calls the subprocess.run interface to execute a sequence of adb commands. For example, it first pushes the test tool binary file under the tools directory to the / data / local / tmp path of the terminal under test via adb push, and then grants executable permissions and starts the tool via adb shell, thereby performing functional verification or resource consumption monitoring of the IDPS process running on the terminal under test. During the test, the control host can also execute commands such as adb shell ps | grep ids_process in real time through the ADB session to detect the liveness status of the IDPS process on the terminal under test. It's important to know that standard test items include file monitoring tests, system resource monitoring tests, abnormal process tests, network card traffic monitoring tests, network connection monitoring tests, and log content monitoring.

[0032] The file monitoring test steps are as follows: 1. Prepare the files and directories to be tested: Specify a readable and writable directory as our test directory, inform IDS of this directory so that it can monitor the directory, and configure the corresponding file monitoring events (CRUD operations, CRUD operations, and move operations).

[0033] 2. Execute the corresponding file operation commands: Add: Create a file in the corresponding directory; Delete: Delete a file in the corresponding directory; Modify: Modify the contents of a file in the corresponding directory; Query: View the contents of a file; Move: Move a file from one directory to another.

[0034] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding event has been recorded.

[0035] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0036] System resource monitoring test steps: 1. Set resource thresholds: First, manually set a CPU / memory threshold and inform the IDS so that it can monitor the corresponding threshold.

[0037] 2. Execute commands: Use commands to artificially increase the system's CPU / memory usage, causing it to exceed the threshold.

[0038] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding event has been recorded.

[0039] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0040] Abnormal process testing steps: 1. Prepare the process to be tested: Specify a process to be tested, check the process's whitelist to ensure that the process is not in the whitelist, and inform IDS of the updated whitelist so that it can monitor the processes in the whitelist.

[0041] 2. Start abnormal process: Start the process to be tested.

[0042] 3. Obtain IDS monitoring results: Access the IDS event log to check if the creation event of the corresponding process has been recorded.

[0043] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0044] Network card traffic monitoring test steps: 1. Set network card thresholds: First, manually set a network card to be monitored, as well as the incoming and outgoing traffic thresholds for the network card, and inform the IDS so that it can monitor the corresponding network card and thresholds.

[0045] 2. Execute commands: Use commands to generate additional traffic, increasing the traffic entering and leaving the network and causing it to exceed the threshold.

[0046] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding event has been recorded.

[0047] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0048] Network connectivity monitoring test: 1. Prepare the process to be tested: Specify a process that can create TCP (Transmission Control Protocol) connections, check the network connection whitelist to ensure that the process is not on the whitelist, and inform IDS of the updated whitelist so that it can monitor the processes on the whitelist.

[0049] 2. Start the abnormal process: Start the process to be tested. 3. Obtain IDS monitoring results: Access the IDS event log to check if the connection events of the corresponding process are recorded.

[0050] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0051] Log content monitoring test: 1. Set up the corresponding logs and content: First, manually specify a log directory to be monitored and the content to be monitored, and inform IDS.

[0052] 2. Modify file: Use commands to write specified content to a specified log file.

[0053] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding event has been recorded.

[0054] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0055] In another specific embodiment, when the attack execution method is a network attack test, the attacking host is controlled through the first communication connection to generate target attack data and send the target attack data to the terminal under test to perform a network attack test on the terminal under test. Specifically, when it is determined that the attack execution method of the current use case is a network attack test, that is, when the use case is located in the cases / network traffic monitoring / general directory, the control host manipulates the attacking host to launch an attack through an established SSH remote control session. Specifically, the control host calls the exec_command method of the SSHClient object in the paramiko library to send the pre-constructed network attack command as a string to the attacking host for execution. For example, the command hping3 -S -p 80 --flood 192.168.225.1 is sent to the attacking host, instructing the attacking host to launch a SYN (Synflooding) flood attack on the IP address of the terminal under test in the wireless LAN; or the command nmap -sS -p 1-1000 192.168.225.1 is sent to perform a port scan on the terminal under test. The attacking host's Linux operating system natively executes these commands, generating target attack packets and sending them directly to the tested terminal via Wi-Fi to stress or functionally test its network intrusion detection module. It's important to know that CAN testing includes: CAN replay attacks, CAN flooding attacks, CAN reverse frame attacks, and CAN obfuscation attacks.

[0056] CAN attack test steps: 1. Create a DBC (Database Can) file: Manually create a DBC file using a script, ensure that the message to be tested is in the DBC file, record the period / length of the message, and finally inform IDS of the updated DBC file so that it can load the new DBC file.

[0057] 2. Sending attack messages: Based on the correct message period / length, manually modify the message period or length to make it send through the driver interface according to the modified configuration, and ensure a high CAN load rate.

[0058] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding CAN attack event is recorded.

[0059] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0060] In the third specific embodiment, when the attack execution method is a bus attack test, abnormal CAN messages are directly sent to the terminal under test through the second communication connection established by the CAN bus to perform a bus attack test on the terminal under test. Specifically, when it is determined that the attack execution method of the current test case is a bus attack test, the control host calls the interface encapsulated in can_api.py in the libs directory during the test case execution phase. This interface further calls the VCI_Transmit function in the ControlCANFD_x64.dll dynamic link library through the ctypes library. According to the baud rate and CANFD enable status of the CAN0 or CAN1 channel defined in the can_cfg field of the configuration file, it constructs and sends abnormal CAN identifier messages outside the whitelist. For example, it sends Fuzz messages with random identifiers, replay attack messages that exceed the range of the whitelist ["0x123","0x765","0x18da00f1"], or malformed messages with data field lengths exceeding the normal range to perform an attack test on the CAN bus intrusion detection module of the terminal under test. It's important to know that network attack testing includes: cross-domain access detection, ICMP smurf attack detection, TCP LAN denial-of-service attack detection, and TCP SYN flood attack detection.

[0061] Network attack testing steps: 1. Generate test commands: Generate corresponding commands according to specific test items; for example, cross-domain access detection, create a network connection to access the public network; TSP SYN flood attack, create a large number of TCP SYN packets.

[0062] 2. Sending attack messages: Log in to the Linux computer via SSH, execute the corresponding attack commands, and transmit the commands to the terminal under test via the WIFI network.

[0063] 3. Obtain IDS monitoring results: Access the IDS event log to check if the corresponding attack event has been recorded.

[0064] 4. Save results to reports: Compare expected results with actual results and save them to reports.

[0065] This integrates various attack test types, enabling unified invocation and avoiding frequent manual switching of types by operators during testing.

[0066] Step S14: Obtain the test result corresponding to the current test case from the terminal under test. After the current test case is executed, determine a new current test case based on the preset order, and continue to execute the step of dynamically loading the test script corresponding to the current test case in the test case list until the test cases in the test case group are executed, and generate the test report corresponding to the target test task.

[0067] In this embodiment, log files are retrieved from the terminal under test using Android Debug Bridge (ADB) commands. The log files are parsed to obtain the test results of the intrusion detection and prevention system in the current test case's attack test. These test results are compared with preset expected test results. If the test results match the preset expected test results, the attack test for the current test case is deemed successful, and the step of dynamically loading the test script corresponding to the current test case from the test case list continues. If the test results do not match the preset expected test results, the attack test for the current test case is deemed unsuccessful. Failure information for the current test case is recorded, and a corresponding test result is generated based on the failure information. Then, a new current test case is determined, and the step of dynamically loading the test script corresponding to the current test case from the test case list continues.

[0068] Specifically, the control host, through an established ADB debugging session, calls the `subprocess.run` interface to execute the command `adbpull / mnt / mnt_share / securityd / log / . / tmp / logs / `, pulling the daily log files stored in the security daemon log directory of the tested terminal's read-only storage partition to the control host's `tmp` temporary working directory. Subsequently, the control host reads the log file content and, based on predefined feature strings in the current test case script—such as alarm keywords like "Flood attack detected" for SYN flood attacks or "CAN ID out of whitelist" for CAN fuzz attacks—calls Python's `re.search` function for regular expression matching. If a match is found, the control host further extracts fields such as the timestamp, attack type, and source address of the log record, generating a structured test result object.

[0069] After obtaining the test results, the control host compares them with the preset expected results in the current test case script. These preset expected results are defined in the member attributes of the test case class; for example, `self.expected_alert = True` indicates that an alarm should be generated, or `self.expected_block = True` indicates that a block should be triggered. If the actual test results match the preset expected results, the control host calls the report logging function in `test_output.py` under the `libs` directory to mark the execution status of the test case as passed. Then, it automatically determines the next test case as the new current test case according to the index order of the test case list and continues to return to the steps of dynamically loading and running the next test case script. If the actual test results do not match the preset expected results—for example, an alarm should be generated but no corresponding record is found in the log, or a block should be triggered but network connectivity tests show that data packets are still reachable—the control host marks the execution status of the test case as failed. Simultaneously, it collects the failure information of the current test case, including the test case name, the location of the failed step, a description of the difference between the actual and expected results, and the current log fragment. This failure information is then encapsulated and saved by calling the report logging function. After recording is complete, the control host still determines the next test case as the new current test case according to the established order and continues to execute the subsequent test process, without interrupting the entire test task due to the failure of a single test case.

[0070] In addition, throughout the entire test execution process, an independent report listening process runs synchronously on the control host. This process collects execution information and test results for each test case in the test case group; it then renders this information and results into a preset Hypertext Markup Language template, summarizing them into a test report for the target test task. Specifically, a TCP socket is created by calling the socket module in the Python standard library and bound to a specified port of the local loopback address for listening. Whenever a test case completes execution and generates a result, the control host's test master process immediately serializes and sends the test case name, test case group, execution result, timestamp, and detailed step information to the report listening process via this socket. The report listening process stores the received data into a result set in memory and updates multiple execution results from the same test case with the latest data. When the control host has traversed all test case groups and the test master process sends a signal that all test cases have been executed, the tester terminates the report listening process via keyboard interrupt. At this point, before exiting, the report monitoring process calls the `Environment` object of the Jinja2 template engine, loads the `report_html.jinja` hypertext markup language template in the `template` directory, passes all test case execution information and judgment results collected in memory as template variables, performs rendering operations, generates a complete hypertext markup language document, writes it to the `output` directory with the current system time as the filename, for example, `xxxx_xx_xx_xx_xx_00_report.html`, and finally obtains a complete summary test report of this target test task. The report monitoring process is shown below: <!-- Report Title --> <h1 class="report-title"> {{cases_info.project_name}} IDPS Automated Test Report {{cases_info.version}}< / h1> <!-- 1. Basic Test Information --> <h2 class="sub-title"> I. Basic Test Information< / h2>

[0071] Figure 3

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Figure 4

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088] Test Project IDPS Intrusion Prevention System Test version {{cases_info.version}} Test type Functional + Automated Testing Test environment {{cases_info.environment}} As described above, in this embodiment, the control host first obtains and parses the preset configuration file of the target test task, extracting test configuration parameters, test case groups, and a detailed list of test cases within each group. Based on these configuration parameters, it automatically establishes a collaborative test environment between the host, the attacking host, and the terminal under test. Subsequently, it initializes each test case group in a preset order. For the current test case in each test case group, it loads its corresponding test script using dynamic loading technology to execute the test task. During execution, based on the test type specified by the current test case, it determines the corresponding attack execution method and initiates an attack test corresponding to the test case to the terminal under test through a preset attack path. After the attack is completed, it obtains the test results from the terminal under test and determines the execution status of the current test case. After the current test case is completed, it determines the next new test case in sequence and repeats the above dynamic loading and execution steps until all test cases in the test case group and even the entire test task are executed, and finally, a complete test report is generated. In this way, through configuration-driven and dynamic script loading, the entire process of test task orchestration is automated, significantly reducing manual intervention. Meanwhile, by coordinating the control of the attack host and bus interface, it supports unified scheduling of multiple types of attacks, including network and bus attacks, effectively improving the efficiency and versatility of automated testing.As shown in the figure, this embodiment of the invention discloses an automated testing device for an intrusion detection and prevention system, applied to the control host of the intrusion detection and prevention system. The testing system of the intrusion detection and prevention system consists of the control host, an attacking host, and a terminal under test. The control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method. The attacking host is connected to the terminal under test via a network. The device includes: an environment establishment module 11, used to acquire and parse a preset configuration file of the target test task to obtain test configuration parameters, test case groups, and a list of test cases in each test case group, and to establish a target test environment according to the test configuration parameters; a script loading module 12, used to initialize each test case group according to a preset order and dynamically load the test script corresponding to the current test case in the test case list; and a test initiation module 13, used to initiate the test according to the current test case group. The test type of the example determines the attack execution method. According to the attack execution method, an attack test corresponding to the current test case is initiated to the terminal under test through a preset method. The report generation module 14 is used to obtain the test result corresponding to the current test case from the terminal under test. After the current test case is executed, a new current test case is determined based on the preset order. The step of dynamically loading the test script corresponding to the current test case in the test case list is continued until the test cases in the test case group are executed, and a test report corresponding to the target test task is generated. Specifically, the test initiation module is used to directly test the terminal under test through the first communication method when the attack execution method is a preset normal test. When the attack execution method is a bus attack test, it directly sends an abnormal CAN message to the terminal under test through the first communication method to perform a bus attack test on the terminal under test. As can be seen from the above, the control host of this application first obtains and parses the preset configuration file of the target test task, extracts the test configuration parameters, test case group and detailed test case list within the group, and automatically establishes a collaborative test environment between the host and the terminal under test based on these configuration parameters. Subsequently, each test case group is initialized sequentially according to a preset order. For the current test case in each test case group, its corresponding test script is loaded using dynamic loading technology to execute the test task. During execution, based on the test type specified by the current test case, the corresponding attack execution method is determined, and an attack test corresponding to that test case is launched to the terminal under test through a preset attack path.After the attack is completed, the test results are retrieved from the tested terminal, and the execution status of the current test case is determined. Once the current test case has finished executing, a new test case is sequentially identified, and the above dynamic loading and execution steps are repeated until all test cases in the test case group and even the entire test task have been executed. Finally, a complete test report is generated. In this way, through configuration-driven and dynamic script loading, the entire process of test task orchestration is automated, significantly reducing manual intervention. Simultaneously, by coordinating the control of the attack host and bus interface, unified scheduling of multiple types of attacks, including network and bus attacks, is supported, effectively improving the efficiency and versatility of automated testing. In some specific implementations, the test configuration parameters include attack host connection parameters, tested terminal connection parameters, test case group configuration parameters, and test environment configuration parameters; correspondingly, the environment establishment module 11 may specifically include: a first communication connection establishment unit, used to establish a first communication connection with the attack host through the Secure Shell protocol, the user's login information, and the attack host connection parameters; a second communication connection establishment unit, used to establish a second communication connection with the tested terminal based on the tested terminal connection parameters, and a USB interface and / or a CAN bus; a third communication connection establishment unit, used to connect the attack host and the tested terminal by constructing a network hotspot; and an environment building unit, used to build a target test environment based on the first communication connection, the second communication connection, and the network hotspot. In some specific embodiments, the second communication connection establishment unit may specifically include: a parameter determination subunit, used to determine CAN bus configuration parameters from the test configuration parameters; wherein, the CAN bus configuration parameters include the baud rate of the CAN channel of the controller area network, whether CAN FD is enabled, and any one or more of the whitelist message identifier IDs; a connection establishment subunit, used to generate corresponding CAN database communication rules according to the whitelist message identifier IDs, so as to establish the second communication connection based on the communication rules. In some specific embodiments, the script loading module 12 may specifically include: an environment establishment unit, used to perform group-level initialization operations on the test case group to establish the execution environment of the test case group; a test case determination unit, used to traverse the test case group to determine the current test case based on a preset order; a script determination unit, used to determine the test script according to the script path corresponding to the current test case; and a test case loading unit, used to call the execution interface of the test script to load the current test case.In some specific implementations, the test initiation module 13 may specifically include: a first terminal testing unit, used to directly test the terminal under test through the second communication connection established by the USB interface when the attack execution mode is a preset conventional test; a second terminal testing unit, used to control the attack host to generate target attack data through the first communication connection and send the target attack data to the terminal under test to perform a network attack test on the terminal under test when the attack execution mode is a network attack test; and a third terminal testing unit, used to directly send abnormal CAN messages to the terminal under test through the second communication connection established by the CAN bus to perform a bus attack test on the terminal under test when the attack execution mode is a bus attack test. In some specific implementations, the report generation module 14 may specifically include: a test result acquisition unit, used to pull log files from the terminal under test via Android Debug Bridge (ADB) commands, parse the log files, and obtain the test results of the intrusion detection and prevention system in the current test case's attack test; a result comparison unit, used to compare the test results with preset test expected results; a first result determination unit, used to determine that the attack test of the current test case has passed if the test results are consistent with the preset test expected results, and continue to execute the step of dynamically loading the test script corresponding to the current test case in the test case list; a second result determination unit, used to determine that the attack test of the current test case has failed if the test results are inconsistent with the preset test expected results, record the failure information of the current test case, generate corresponding test results based on the failure information, then determine a new current test case, and continue to execute the step of dynamically loading the test script corresponding to the current test case in the test case list. In some specific embodiments, the report generation module 14 may specifically include: a result collection unit, used to collect the execution information and test judgment results of each test case in the test case group through a preset report monitoring process; and a report generation unit, used to render the execution information and test judgment results into a preset hypertext markup language template and summarize them into a test report for the target test task. Furthermore, this application also discloses an electronic device, which is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the automated testing method of the intrusion detection and prevention system disclosed in any of the foregoing embodiments.In addition, the electronic device 20 in this embodiment can specifically be an electronic computer. In this embodiment, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here. In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc., and the resources stored therein can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage. Among them, the operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the automated testing method of the intrusion detection and prevention system executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks. Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned automated testing method for an intrusion detection and prevention system. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to the method section. Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both.Software modules may be housed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An automated testing method for an intrusion detection and prevention system, characterized in that, A control host for an intrusion detection and prevention system, wherein the test system of the intrusion detection and prevention system comprises the control host, an attacking host, and a terminal under test; the control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method, and the attacking host is connected to the terminal under test via a network; the method includes: Obtain and parse the preset configuration file of the target test task to obtain test configuration parameters, test case groups and test case lists in each test case group, and establish the target test environment according to the test configuration parameters; The test case groups are initialized according to a preset order, and the test scripts corresponding to the current test case in the test case list are dynamically loaded. The attack execution method is determined according to the test type of the current test case, and the attack test corresponding to the current test case is initiated to the terminal under test according to the attack execution method through a preset method. The test results corresponding to the current test case are obtained from the terminal under test. After the current test case is executed, a new current test case is determined based on the preset order. The step of dynamically loading the test script corresponding to the current test case in the test case list is continued until the test cases in the test case group are executed and the corresponding test report of the target test task is generated. The step of determining the attack execution method based on the test type of the current test case, and initiating an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method, includes: When the attack execution method is a preset routine test, the terminal under test is directly tested through the first communication method. When the attack execution method is a bus attack test, an abnormal CAN message is sent directly to the terminal under test through the first communication method to perform a bus attack test on the terminal under test.

2. The automated testing method for an intrusion detection and prevention system according to claim 1, characterized in that, The test configuration parameters include attack host connection parameters, tested terminal connection parameters, test case group configuration parameters, and test environment configuration parameters. Accordingly, establishing the target test environment based on the test configuration parameters includes: A first communication connection is established between the attacker and the host through the Secure Shell protocol, the user's login information, and the attacker's host connection parameters. A second communication connection is established between the terminal under test and the terminal under test based on the connection parameters of the terminal under test, as well as the USB interface and / or CAN bus. The attacking host and the tested terminal are connected by creating a network hotspot; The target test environment is built based on the first communication connection, the second communication connection, and the network hotspot.

3. The automated testing method for an intrusion detection and prevention system according to claim 2, characterized in that, The establishment of a second communication connection with the terminal under test based on the connection parameters of the terminal under test, and the USB interface and / or CAN bus, includes: The CAN bus configuration parameters are determined from the test configuration parameters; wherein, the CAN bus configuration parameters include the baud rate of the CAN channel of the controller area network, whether the CAN FD parameter is enabled, and any one or more of the whitelist message identifier IDs; Based on the whitelist message identifier ID list, corresponding communication rules for the CAN database are generated to establish the second communication connection based on the communication rules.

4. The automated testing method for an intrusion detection and prevention system according to claim 3, characterized in that, The initialization operation of each test case group based on a preset order, and the dynamic loading of the test script corresponding to the current test case in the test case list, includes: Perform group-level initialization operations on the test case group to establish the execution environment for the test case group; The test case group is traversed to determine the current test case based on a preset order; The test script is determined based on the script path corresponding to the current test case; The execution interface of the test script is invoked to load the current test case.

5. The automated testing method for an intrusion detection and prevention system according to claim 4, characterized in that, The step of determining the attack execution method based on the test type of the current test case, and initiating an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method, includes: When the attack execution method is a preset routine test, the terminal under test is directly tested through the second communication connection established through the USB interface. When the attack execution method is a network attack test, the attack host is controlled to generate target attack data through the first communication connection and send the target attack data to the terminal under test to perform a network attack test on the terminal under test. When the attack execution method is a bus attack test, an abnormal CAN message is sent directly to the terminal under test through the second communication connection established by the CAN bus to perform a bus attack test on the terminal under test.

6. The automated testing method for an intrusion detection and prevention system according to claim 1, characterized in that, The step of obtaining the test result corresponding to the current test case from the terminal under test, and determining a new current test case based on the preset order after the current test case has been executed, includes: Log files are retrieved from the terminal under test using the Android Debug Bridge ADB command, and the log files are parsed to obtain the test results of the intrusion detection and prevention system in the attack test of the current test case. Compare the test results with the preset test expectations; If the test result is consistent with the preset test expectation result, it is determined that the attack test of the current test case has passed, and the step of dynamically loading the test script corresponding to the current test case in the test case list continues to be executed; If the test result is inconsistent with the preset test expectation result, it is determined that the attack test of the current test case has failed, the failure information of the current test case is recorded, the corresponding test result is generated based on the failure information, a new current test case is determined, and the step of dynamically loading the test script corresponding to the current test case in the test case list continues to be executed.

7. The automated testing method for an intrusion detection and prevention system according to claim 1, characterized in that, The generation of the test report corresponding to the target test task includes: By using a pre-defined report monitoring process, the execution information and test judgment results of each test case in the test case group are collected. The execution information and test results are rendered into a preset hypertext markup language template and summarized into a test report for the target test task.

8. An automated testing device for an intrusion detection and prevention system, characterized in that, A control host for an intrusion detection and prevention system, wherein the testing system of the intrusion detection and prevention system comprises the control host, an attacking host, and a terminal under test; the control host is connected to the terminal under test via a first communication method and to the attacking host via a second communication method, and the attacking host is connected to the terminal under test via a network; the device includes: The environment setup module is used to obtain and parse the preset configuration file of the target test task to get the test configuration parameters, test case groups and test case lists in each test case group, and to set up the target test environment according to the test configuration parameters. The script loading module is used to perform initialization operations on each of the test case groups based on a preset order, and dynamically load the test script corresponding to the current test case in the test case list; The test initiation module is used to determine the attack execution method according to the test type of the current test case, and to initiate an attack test corresponding to the current test case to the terminal under test according to the attack execution method through a preset method. The report generation module is used to obtain the test results corresponding to the current test case from the terminal under test, determine a new current test case based on the preset order after the current test case is executed, and continue to execute the step of dynamically loading the test script corresponding to the current test case in the test case list until the test cases in the test case group are executed, and generate the test report corresponding to the target test task. Specifically, the test initiation module is used to directly test the terminal under test through the first communication method when the attack execution method is a preset normal test; and to directly send an abnormal CAN message to the terminal under test through the first communication method when the attack execution method is a bus attack test.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement an automated testing method for an intrusion detection and prevention system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the automated testing method for the intrusion detection and prevention system as described in any one of claims 1 to 7.

Images