Test method, device and equipment of vehicle-mounted controller and storage medium
By automating the acquisition and comparison of security event logs, the problem of low testing efficiency of vehicle-mounted controller IDS was solved, the reliability and reproducibility of test results were achieved, and the comprehensive protection capability and performance evaluation of IDS were ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SAIC GM WULING AUTOMOBILE CO LTD
- Filing Date
- 2025-12-22
- Publication Date
- 2026-05-01
AI Technical Summary
In the existing technology, the functional testing of intrusion detection system (IDS) of vehicle controllers relies on manual operation, which leads to low testing efficiency, inability to achieve rapid and large-scale test iterations, inconsistent test conditions, difficulty in reliable reproduction, and inability to accurately evaluate performance and regression testing.
By automating the acquisition of test cases, injecting attack traffic, and acquiring and comparing security event logs in real time, the entire process of test cases is automated, ensuring that the test conditions are consistent each time. The program precisely controls the attack parameters, automatically acquires and compares the security event log set, and determines the test results.
It improves testing efficiency, ensures the reliability and reproducibility of test results, provides a solid foundation for performance evaluation and problem localization, and can systematically cover complex attack scenarios to verify the comprehensive protection capabilities of IDS.
Smart Images
Figure CN121956940A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle technology, and more specifically to a testing method, apparatus, device, and storage medium for an on-board controller. Background Technology
[0002] With the development of intelligent and connected vehicles, the vehicle controller has gradually become the core processing unit of the vehicle, responsible for processing a large amount of network data within the vehicle. To cope with the increasingly severe cyberattack threats, intrusion detection systems (IDS) are usually integrated into vehicles as a critical security component, which can be used to monitor and warn of malicious attack behaviors related to the vehicle controller in real time.
[0003] In practical applications, after integrating an IDS into a vehicle, functional testing is typically required to verify its functionality and reliability. Currently, functional verification of integrated IDS mainly relies on manual testing. Specifically, testers use functional networking tools to manually simulate attack traffic against the vehicle controller and manually observe whether the IDS generates the expected alarm logs.
[0004] However, since the execution of test cases and the collection and comparison of logs rely entirely on manual operation, it may not be possible to achieve rapid and large-scale test iterations and it is difficult to adapt to the rapid iteration pace of software development, which may lead to low testing efficiency. In addition, manual operation cannot precisely control the timing, frequency and quantity of attack traffic, which may lead to inconsistent test conditions, unreliable test results, and difficulty in using them for accurate performance evaluation and regression testing.
[0005] It should be noted that the information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention
[0006] This application provides a testing method, apparatus, device, and storage medium for vehicle controllers, in order to solve the problem in related technologies where manual testing of vehicle controllers by testers may lead to low testing efficiency and unreliable reproducibility.
[0007] In a first aspect, embodiments of this application provide a testing method for an on-board controller, comprising: Obtain test cases corresponding to the intrusion detection system. The test cases include attack vector parameters and expected security event logs corresponding to the attack vector parameters. The intrusion detection system is used to monitor abnormal behavior in the communication of the vehicle controller. According to the attack vector parameters, attack traffic corresponding to the attack vector parameters is injected into the vehicle controller, and the attack traffic is used to characterize the attack behavior against the vehicle controller; During the process of injecting the attack traffic into the vehicle controller, the actual security event log set corresponding to the intrusion detection system is obtained; Based on the comparison between the expected security event log and the actual security event log set, the test result corresponding to the test case is determined.
[0008] In one possible implementation, injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters includes: Based on the attack vector parameters, determine the attack injection command corresponding to the attack vector parameters; The attack injection command is sent to the attack injection device, and the attack injection device responds to the attack injection command by injecting attack traffic corresponding to the attack vector parameters into the vehicle controller.
[0009] In one possible implementation, obtaining the actual security event log set corresponding to the intrusion detection system includes: A log collection command is sent to a log collection device, which responds to the log collection command and collects all actual security event logs corresponding to the intrusion detection system. Receive the actual security event log set sent by the log collection device, which includes all the actual security event logs.
[0010] In one possible implementation, the actual security event log set includes a correspondence between actual security event logs and the test cases. The step of determining the test result corresponding to the test case based on the comparison between the expected security event logs and the actual security event log set includes: Based on the correspondence between actual security event logs and test cases, target actual security event logs corresponding to the test cases are filtered out from the actual security event log set; Based on the comparison between the expected security event log and the actual security event log of the target, the test result corresponding to the test case is determined.
[0011] In one possible implementation, determining the test result corresponding to the test case based on the comparison between the expected security event log and the target actual security event log includes: Based on the comparison results of the expected security event log and the target actual security event log in each of the multiple comparison dimensions, the test result corresponding to the test case is determined.
[0012] In one possible implementation, before obtaining the test cases corresponding to the intrusion detection system, the method further includes: constructing a test case library corresponding to the intrusion detection system based on the functional information of the intrusion detection system; The step of obtaining test cases corresponding to the intrusion detection system includes: obtaining test cases corresponding to the intrusion detection system from the test case library.
[0013] In one possible implementation, obtaining the test cases corresponding to the intrusion detection system from the test case library includes: obtaining all test cases in the test case library; The step of injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters includes: for each of the test cases in all the test cases, injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters in the test case; The step of determining the test result corresponding to the test case based on the comparison result between the expected security event log and the actual security event log set includes: determining the test result corresponding to each test case based on the comparison result between the expected security event log and the actual security event log set in each test case.
[0014] In one possible implementation, the method further includes: Based on the test results corresponding to each test case, a test report corresponding to the intrusion detection system is generated.
[0015] Secondly, embodiments of this application provide a testing apparatus for an on-board controller, comprising: The test case acquisition module is used to acquire test cases corresponding to the intrusion detection system. The test cases include attack vector parameters and expected security event logs corresponding to the attack vector parameters. The intrusion detection system is used to monitor abnormal behavior in the communication of the vehicle controller. An attack traffic injection module is used to inject attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters. The attack traffic is used to characterize the attack behavior against the vehicle controller. The log set acquisition module is used to acquire the actual security event log set corresponding to the intrusion detection system during the process of injecting the attack traffic into the vehicle controller; The test result determination module is used to determine the test result corresponding to the test case based on the comparison result between the expected security event log and the actual security event log set.
[0016] Thirdly, embodiments of this application provide a control device, including: processor; Memory; And a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, causes the control device to perform the method described in any one of the first aspects.
[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the first aspects.
[0018] In this embodiment, test cases, including attack vector parameters and expected security event logs corresponding to the attack vector parameters, are first obtained. Then, attack traffic corresponding to the attack vector parameters is injected into the vehicle controller. During the injection of attack traffic into the vehicle controller, the actual security event log set corresponding to the intrusion detection system is obtained. Finally, the test result corresponding to the test cases is determined based on the comparison between the expected security event logs and the actual security event log set. It can be understood that automating the entire process of obtaining test cases, attack injection, log collection, and comparison eliminates the need for testers to manually simulate attack traffic using functional network tools and manually observe whether the intrusion detection system generates the expected alarm logs, thereby improving testing efficiency. Furthermore, precise program control of attack parameters ensures that the conditions for each test are completely consistent, making the test results reliable and reproducible. This ensures the accuracy and repeatability of the test, providing a solid foundation for performance evaluation and problem localization. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application.
[0021] Figure 2 This is a flowchart illustrating a testing method for an on-board controller provided in an embodiment of this application.
[0022] Figure 3 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application.
[0023] Figure 4 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application.
[0024] Figure 5 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application.
[0025] Figure 6 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application.
[0026] Figure 7 This is a schematic diagram of the structure of a test device for an on-board controller provided in an embodiment of this application.
[0027] Figure 8 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation
[0028] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0029] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0030] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0031] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0032] See Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1As shown, vehicle 100 includes an onboard controller 101. The onboard controller 101, as the core processing unit of vehicle 100, is responsible for processing a large amount of network data within the vehicle. Therefore, the onboard controller 101 is vulnerable to cyberattacks.
[0033] In order to cope with the increasingly serious cyberattack threats, intrusion detection systems (IDS) are usually integrated into vehicles as a key security component, which can be used to monitor and warn of malicious attack behaviors related to the vehicle controller in real time.
[0034] It should be pointed out that, Figure 1 The vehicle shown is merely an exemplary description and should not be construed as limiting the scope of protection of this application. Of course, this application does not specifically limit the type of on-board controller. For example, on-board controllers include, but are not limited to, vehicle controllers, engine controllers, motor controllers, etc.
[0035] In practical applications, after integrating an IDS into a vehicle, functional testing is typically required to verify its functionality and reliability. Currently, functional verification of integrated IDS primarily relies on manual testing. Specifically, testers use functional networking tools to manually simulate attack traffic and manually observe whether the IDS generates the expected alarm logs.
[0036] However, since the execution of test cases and the collection and comparison of logs rely entirely on manual operation, it may not be possible to achieve rapid and large-scale test iterations and it is difficult to adapt to the rapid iteration pace of software development, which may lead to low testing efficiency. In addition, manual operation cannot precisely control the timing, frequency and quantity of attack traffic, which may lead to inconsistent test conditions, unreliable test results, and difficulty in using them for accurate performance evaluation and regression testing.
[0037] To address the aforementioned issues, this application provides a testing method for an in-vehicle controller. The method first acquires test cases including attack vector parameters and expected security event logs corresponding to the attack vector parameters; secondly, it injects attack traffic corresponding to the attack vector parameters into the in-vehicle controller; then, during the injection of attack traffic into the in-vehicle controller, it acquires an actual security event log set corresponding to the intrusion detection system; finally, based on the comparison between the expected security event logs and the actual security event log set, it determines the test results corresponding to the test cases.
[0038] Understandably, automating the entire process of acquiring test cases, injecting attacks, collecting and comparing logs eliminates the need for testers to manually simulate attack traffic using functional network tools and observe whether the intrusion detection system generates the expected alarm logs, thus improving testing efficiency. Furthermore, precise program control of attack parameters ensures that the conditions are completely consistent for each test, making the test results reliable and reproducible. This guarantees the accuracy and repeatability of the tests, providing a solid foundation for performance evaluation and problem localization.
[0039] Specifically, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments.
[0040] See Figure 2 This is a flowchart illustrating a testing method for an on-board controller provided in an embodiment of this application. This method can be applied to... Figure 1 In the application scenarios shown, such as Figure 2 As shown, it specifically includes steps S201 to S204.
[0041] Step S201: Obtain test cases corresponding to the intrusion detection system.
[0042] In this embodiment, the IDS can monitor the network transmission of the vehicle controller in real time and issue an alarm or take proactive measures when it detects unauthorized intrusion into the vehicle controller. In other words, the intrusion detection system can be used to monitor abnormal behavior in the communication of the vehicle controller.
[0043] It should be noted that as long as the IDS can monitor abnormal behavior in the communication of the vehicle controller in real time, the IDS can be integrated into any other device or system in the vehicle, such as the vehicle controller, vehicle monitoring system, etc. This application does not impose specific limitations in this regard.
[0044] As mentioned above, after integrating the Intrusion Detection System (IDS) used to monitor the vehicle controller into the vehicle, functional testing of the integrated IDS is usually required to verify its functional effectiveness and reliability. Therefore, it is necessary to first obtain test cases corresponding to the intrusion detection system.
[0045] In the embodiments of this application, test cases typically include a test target, attack vector parameters, and expected security event logs corresponding to the attack vector parameters. It can be understood that a test case can be a data structure containing three key elements: a test target, attack vector parameters, and expected security event logs.
[0046] The test target specifies the IDS functionalities to be verified, such as "CAN BusOff attack detection and rate limiting". The attack vector parameters define all parameters for the attack simulation, such as {Attack Type: "CAN BusOff", Channel: 1, Duration: 60 seconds, Number of Attack IDs: 70}. The expected security event log defines the conditions that the security event logs expected to be output by the IDS should meet, such as {Event ID: 966026440, Maximum number of records: 50, Field validation: [CAN_ID∈(the set of 70 injected IDs), Channel=1]}. It can be understood that the maximum number of records refers to the maximum number of events that can be reported as set by the rate limiting mechanism.
[0047] To obtain test cases corresponding to an intrusion detection system, one possible implementation is for testers to manually set the relevant data for the test cases. However, this method may require a high level of expertise from the testers and is prone to errors. Furthermore, manual testing struggles to effectively and thoroughly verify advanced strategies within the intrusion detection system, potentially leading to low testing efficiency.
[0048] For example, when CPU utilization is high, the reporting strategy may be adjusted. Manual testing may not be able to simulate high CPU utilization to test the degradation strategy of the intrusion detection system, which may result in low testing efficiency.
[0049] See Figure 3 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application. Figure 3 As shown, the embodiments of this application are in Figure 2 Based on the embodiment shown, step S301 is included before step S201, and step S201 specifically includes step S302.
[0050] Step S301: Based on the functional information of the intrusion detection system, construct a test case library corresponding to the intrusion detection system.
[0051] Understandably, the functional information of an intrusion detection system (IDS) is used to characterize its capabilities, such as monitoring network traffic, detecting abnormal behavior, generating security alerts, and recording event logs. Furthermore, key information can be extracted and analyzed from the IDS's functional information to generate a test case library corresponding to the IDS.
[0052] In the embodiments of this application, the test case library typically includes multiple test cases corresponding to the intrusion detection system. For example, test cases corresponding to the time window-based rate limiting mechanism (e.g., reporting a maximum of 50 events within 1 minute), test cases corresponding to the anti-duplicate reporting mechanism (e.g., not recording the same event repeatedly within 10 seconds), and test cases corresponding to the resource awareness strategy (e.g., adjusting the reporting strategy when CPU utilization is high), etc.
[0053] Furthermore, test cases can be automatically retrieved from the test case library to test the intrusion detection system used to monitor the vehicle controller.
[0054] Step S302: Obtain the test cases corresponding to the intrusion detection system from the test case library.
[0055] In this embodiment of the application, after the test case library is generated, test cases can be obtained from the test case library to test the intrusion detection system used to monitor the vehicle controller.
[0056] Understandably, a structured test case set allows for the systematic design and execution of hundreds or thousands of test scenarios without requiring testers to manually set test case data. This reduces the professional requirements for users to some extent, and is less prone to errors, thereby improving testing efficiency.
[0057] Step S202: Based on the attack vector parameters, inject the attack traffic corresponding to the attack vector parameters into the vehicle controller.
[0058] Understandably, attack traffic is used to characterize attack behavior against the vehicle controller. In this embodiment, after obtaining test cases, the attack traffic corresponding to the attack vector parameters in the test cases can be determined, and the attack traffic can be injected into the vehicle controller to simulate attack behavior against the vehicle controller.
[0059] For example, the attack vector parameters are: {Attack type: "CAN BusOff", channel: 1, duration: 60 seconds, number of attack IDs: 70}, and the corresponding attack traffic is: control CANoe to inject 70 messages with different IDs into CAN channel 1 within 60 seconds to trigger BusOff.
[0060] It should be noted that the attack traffic injected into the vehicle controller includes, but is not limited to, CAN attack traffic and Ethernet attack traffic.
[0061] In practical applications, the control device can inject attack traffic corresponding to the attack vector parameters into the vehicle controller. To reduce the data processing burden on the control device, the function of injecting attack traffic into the vehicle controller can be separated from the control device.
[0062] See Figure 4 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application. Figure 4 As shown, the embodiments of this application are in Figure 2 Based on the illustrated embodiment, step S202 specifically includes steps S401-S403.
[0063] Step S401: The control device determines the attack injection command corresponding to the attack vector parameters based on the attack vector parameters.
[0064] In this embodiment, the attack injection instruction includes attack vector parameters. After the control device obtains the test cases, it can generate an attack injection instruction including the attack vector parameters. It can be understood that the attack injection instruction is used to instruct the attack injection device to inject an attack vector corresponding to the attack vector parameters into the vehicle controller.
[0065] It should be noted that the control equipment is used to run the test management software and serves as the control center for the test process. Control equipment includes, but is not limited to, personal laptops, mobile phones, desktop computers, tablets, and personal computers (PCs).
[0066] Step S402: The control device sends an attack injection command to the attack injection device.
[0067] In this embodiment of the application, after the control device generates an attack injection command, it can send the attack injection command to the attack injection device, so that the attack injection device injects an attack vector corresponding to the attack vector parameters into the vehicle controller.
[0068] In practical applications, there are often attack injection devices for different types of network attacks. Each type of attack injection device can typically accurately simulate the attack traffic corresponding to the type of network attack. For example, CANoe hardware can typically accurately simulate CAN attack traffic; Ethernet traffic generators can typically accurately simulate Ethernet attack traffic.
[0069] Therefore, to improve the accuracy of attack traffic simulation and thus enhance testing precision, in one possible implementation, the control device can send an attack injection command to the attack injection device corresponding to the command description information. It can be understood that the command description information corresponding to the attack injection command specifies the interface sending the attack injection command and the attack injection device receiving the command.
[0070] In this embodiment, the instruction description information typically corresponds to the test target. To facilitate obtaining the instruction description information corresponding to the attack injection instruction, the correspondence between instruction description information and test targets can be pre-stored in the relevant storage module of the control device. During the process of the control device generating the attack injection instruction corresponding to the attack vector parameters, the instruction description information corresponding to the generated attack injection instruction is determined based on the correspondence between the instruction description information and the test target. This determines the interface for sending the attack injection instruction, and the attack injection instruction is then sent to the corresponding attack injection device through this interface.
[0071] Step S403: In response to the attack injection command, the attack injection device injects attack traffic corresponding to the attack vector parameters into the vehicle controller.
[0072] Specifically, when the attack injection device receives an attack injection command from the control device, it can parse the attack vector parameters from the command. Furthermore, the attack injection device can determine the corresponding attack traffic based on the attack vector parameters and inject that attack traffic into the vehicle controller, thereby simulating an attack on the vehicle controller.
[0073] In this embodiment, the function of injecting attack traffic into the vehicle controller is separated from the control device, and the attack injection device is responsible for the attack injection function. The intrusion detection system used to monitor the vehicle controller is tested through the collaborative work of the control device and the attack injection device. The two cooperate with each other and are logically separated. While realizing the testing of the intrusion detection system used to monitor the vehicle controller, the logic of the control device is simplified and the function is clearly defined, which reduces the data processing burden of the control device to a certain extent.
[0074] Step S203: During the process of injecting attack traffic into the vehicle controller, obtain the actual security event log set corresponding to the intrusion detection system.
[0075] Understandably, the actual security event log set is used to record the security event logs actually output by the intrusion detection system. It typically includes actual security event logs corresponding to multiple abnormal behaviors of the vehicle controller. The actual security event log for each abnormal behavior records the specific behavior or abnormal event that caused the vehicle controller to exhibit that abnormal behavior.
[0076] To verify the functionality and reliability of the intrusion detection system, in this embodiment of the application, the actual security event log set corresponding to the intrusion detection system can be obtained during the process of injecting attack traffic into the vehicle controller. This allows for analysis of the actual security event log set and further determination of the test results corresponding to the test cases.
[0077] In practical applications, control devices can continuously monitor and capture the actual security event logs corresponding to the intrusion detection system. To reduce the data processing burden on the control devices, the function of collecting actual security event logs can be separated from the control devices.
[0078] See Figure 5 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application. Figure 5 As shown, the embodiments of this application are in Figure 4 Based on the illustrated embodiment, step S203 specifically includes steps S501-S504.
[0079] Step S501: The control device sends a log collection command to the log collection device.
[0080] In this application, while the control device sends an attack injection command to the attack injection device, causing the attack injection device to inject attack traffic into the vehicle controller, it can also send a log collection command to the log collection device, thereby enabling the log collection device to collect the actual security event logs corresponding to the intrusion detection system.
[0081] Understandably, the log collection command is used to instruct the log collection device to collect all actual security event logs corresponding to the intrusion detection system.
[0082] Step S502: In response to the log collection command, the log collection device collects all actual security event logs corresponding to the intrusion detection system during the process of injecting attack traffic into the vehicle controller.
[0083] In this embodiment of the application, when the log collection device receives the log collection instruction sent by the control device, while the attack injection device injects attack traffic into the vehicle controller, the log collector can collect all security event logs generated by the intrusion detection system in real time through the vehicle diagnostic interface or network interface, that is, the actual security event logs.
[0084] Step S503: The log collection device sends a set of actual security event logs, including all actual security event logs, to the control device.
[0085] In this embodiment of the application, after the log collection device collects all security event logs generated by the intrusion detection system, it can send an actual security event log set, which includes all actual security event logs, to the control device, so that the control device can analyze the actual security event log set and thus determine the test results corresponding to the test cases.
[0086] Step S504: The control device receives the actual security event log set, which includes all actual security event logs, sent by the log collection device.
[0087] In this embodiment of the application, when the log collection device sends an actual security event log set including all actual security event logs to the control device, the control device is able to receive the actual security event log set.
[0088] In this embodiment, the function of collecting actual security event logs is separated from the control device, and the log collection device is responsible for log collection. The intrusion detection system used to monitor the vehicle controller is tested through the collaborative work of the control device, the attack injection device, and the log collection device. The three work together with logical separation, achieving the testing of the intrusion detection system used to monitor the vehicle controller while simplifying the logic of the control device and clearly defining its functions, thus reducing the data processing burden on the control device to some extent.
[0089] Step S204: Based on the comparison results between the expected security event log and the actual security event log set, determine the test results corresponding to the test cases.
[0090] As mentioned above, the expected security event log is used to define the conditions that the security event logs expected to be output by the IDS should meet. Therefore, the actual security event log set can be analyzed to compare whether the actual security event logs generated by the intrusion detection system meet the conditions in the expected security event log.
[0091] However, not all actual security event logs in the actual security event log set correspond to the current test case. Therefore, analyzing every actual security event log in the actual security event log set not only wastes testing time but also increases the data processing load on the control device, resulting in low testing efficiency.
[0092] Therefore, in one possible implementation, the target actual security event logs corresponding to the test cases are filtered out from the actual security event log set based on the correspondence between the actual security event logs and the test cases; the test results corresponding to the test cases are determined based on the comparison results between the expected security event logs and the target actual security event logs.
[0093] It is understood that the actual security event log set includes the correspondence between actual security event logs and test cases. For example, the correspondence between actual security event logs and test cases is shown in Table 1. When the test case is identified as Test Case 1, the actual security event log can be identified as Actual Security Event Log 1; therefore, the target actual security event log is the log identified as Actual Security Event Log 1. When the test case is identified as Test Case 2, the actual security event log can be identified as Actual Security Event Log 2; therefore, the target actual security event log is the log identified as Actual Security Event Log 2. And so on, as will not be elaborated further in this embodiment.
[0094] Table 1: Furthermore, after filtering out the target actual security event logs corresponding to the test cases from the actual security event log set, the target actual security event logs can be analyzed to determine whether the target actual security event logs meet the conditions in the expected security event logs corresponding to the test cases.
[0095] To improve the credibility and accuracy of test results, one possible implementation is to determine the test results corresponding to the test cases based on the comparison results of the expected security event logs and the actual security event logs in multiple comparison dimensions.
[0096] Understandably, the comparison dimensions include, but are not limited to, existence comparison, quantity comparison, field matching, and timing checks. Specifically, existence comparison refers to comparing whether the actual security event logs of the target are the events expected to be generated; quantity comparison refers to whether the number of events recorded in the actual security event logs of the target exceeds the expected limit of the rate limiting mechanism, i.e., whether the number of events is less than or equal to the maximum number of records; field matching refers to whether the key fields in the actual security event logs of the target completely match the attack vector parameters, such as attack identifier CAN_ID, attacker IP (i.e., Src_IP), attack count, etc.; timing checks refer to whether the reporting time of the events recorded in the actual security event logs of the target is within the expected detection period.
[0097] In this embodiment, when the expected security event log and the actual target security event log match in every comparison dimension across multiple comparison dimensions, the test result corresponding to the test case is determined to be a successful test. It can be understood that when the actual target security event log is the expected generated event, the number of events is less than or equal to the maximum number of records, the key fields completely match the attack vector parameters, and the event reporting time is within the expected detection period, the IDS function corresponding to the test case can be considered normal. Therefore, the test result corresponding to the test case can be determined to be a successful test.
[0098] Conversely, if the expected security event log and the actual target security event log do not match in any of the comparison dimensions, the test result corresponding to the test case is determined to be a test failure. This can be understood as follows: when the actual target security event log matches the expected events, and / or the number of events is less than or equal to the maximum number of records, and / or the key fields perfectly match the attack vector parameters, and / or the event reporting time is within the expected detection period, the IDS function corresponding to the test case can be considered to have malfunctioned. Therefore, the test result corresponding to the test case can be determined to be a test failure.
[0099] It should be noted that when the test result corresponding to the test case is a test failure, the control device can output details of the difference between the expected security event log and the target actual security event log, so that testers can understand the reason for the test failure in a timely manner.
[0100] In this embodiment of the application, by filtering out the target actual security event logs corresponding to the test cases and filtering out other actual security event logs that are not related to the test cases, the control device can avoid unnecessary log comparison steps, which reduces the test time to a certain extent and reduces the data processing volume of the control device, thereby improving the test efficiency.
[0101] In this embodiment, test cases including attack vector parameters and expected security event logs corresponding to the attack vector parameters are first obtained; secondly, attack traffic corresponding to the attack vector parameters is injected into the vehicle controller according to the attack vector parameters; then, during the process of injecting attack traffic into the vehicle controller, the actual security event log set corresponding to the intrusion detection system is obtained; finally, the test result corresponding to the test cases is determined based on the comparison result between the expected security event logs and the actual security event log set.
[0102] Understandably, automating the entire process of acquiring test cases, injecting attacks, collecting and comparing logs eliminates the need for testers to manually simulate attack traffic using functional network tools and observe whether the intrusion detection system generates the expected alarm logs, thus improving testing efficiency. Furthermore, precise program control of attack parameters ensures that the conditions are completely consistent for each test, making the test results reliable and reproducible. This guarantees the accuracy and repeatability of the tests, providing a solid foundation for performance evaluation and problem localization.
[0103] In practical applications, manual testing is often difficult to systematically cover all predefined and complex attack scenarios (such as multiple attack combinations, traffic attacks of different intensities, etc.), which can easily create test blind spots and fail to guarantee the comprehensive protection capabilities of IDS.
[0104] Therefore, a test case library corresponding to the intrusion detection system can be constructed based on the system's functional information. This library typically includes multiple test cases specific to the intrusion detection system. Furthermore, each test case in the library can be tested.
[0105] See Figure 6 This is a flowchart illustrating another testing method for an on-board controller provided in an embodiment of this application. Figure 6 As shown, the embodiments of this application are in Figure 3 Based on the illustrated embodiment, step S302 includes step S601, step S202 includes step S602, and step S204 includes step S603.
[0106] Step S601: Obtain all test cases in the test case library. In this embodiment of the application, after the test case library is built, all test cases in the test case library can be obtained to perform more comprehensive testing of the intrusion detection system using each test case.
[0107] Step S602: For each test case in all test cases, inject attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters in the test case.
[0108] In this embodiment of the application, for each test case, the attack traffic corresponding to each test case can be determined based on the attack vector parameters in each test case, and the attack traffic can be injected into the vehicle controller to simulate different attack behaviors against the vehicle controller.
[0109] It should be noted that the test case library typically covers, but is not limited to, test scenarios such as CAN-IDS testing, ETH-IDS testing, and communication security testing. Specifically, CAN-IDS testing refers to automated testing that includes bus status anomalies (BusOff) and message length anomalies; ETH-IDS testing refers to automated testing that includes high-frequency connection attacks and TCP SYN Flood attacks; and communication security testing refers to automated testing and replay protection verification for scenarios such as TLS connection establishment failure and diagnostic service authentication failure.
[0110] In addition, during the process of injecting the attack stream corresponding to each test case into the vehicle control system, the control device or log collector can continuously listen to and capture the security event logs corresponding to each test case, thereby enabling the control device to analyze the actual security event log set and determine the test results corresponding to each test case.
[0111] Step S603: Based on the comparison results between the expected security event logs and the actual security event log sets in each test case, determine the test results corresponding to each test case.
[0112] In this embodiment, the actual security event log set can be analyzed to compare whether the actual security event log corresponding to each test case meets the conditions in the expected security event log corresponding to the test case. Specific details related to this embodiment can be found in the description of the above method embodiments; for brevity, they will not be repeated here.
[0113] Furthermore, to enable testers to gain a more comprehensive and detailed understanding of the functionality and effectiveness of the intrusion detection system, once the test results corresponding to each test case are determined, a test report corresponding to the intrusion detection system can be generated based on the test results corresponding to each test case.
[0114] The test report includes, but is not limited to, the overall pass rate, the test results for each test case, and details of the differences between the actual security event logs and the expected security event logs for failed test cases.
[0115] In this embodiment, a structured test case library allows for the systematic design and execution of hundreds or thousands of test scenarios, ensuring comprehensive testing of all declared functions of the intrusion detection system and guaranteeing the completeness and coverage of the tests. Furthermore, it effectively verifies deep-seated IDS strategies that are difficult to access manually, thereby verifying the stability and robustness of the IDS in real-world complex environments.
[0116] Corresponding to the above embodiments, this application also provides a testing device for an on-board controller.
[0117] See Figure 7 This is a schematic diagram of the structure of a test device for an on-board controller provided in an embodiment of this application. Figure 7 As shown, the testing device 700 for the vehicle controller includes a test case acquisition module 701, an attack traffic injection module 702, a log set acquisition module 703, and a test result determination module 704.
[0118] Specifically, the test case acquisition module 701 is used to acquire test cases corresponding to the intrusion detection system. The test cases include attack vector parameters and expected security event logs corresponding to the attack vector parameters. The intrusion detection system is used to monitor abnormal behavior in the communication of the vehicle controller. The attack traffic injection module 702 is used to inject attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters. The attack traffic is used to characterize the attack behavior on the vehicle controller. The log set acquisition module 703 is used to acquire the actual security event log set corresponding to the intrusion detection system during the process of injecting attack traffic into the vehicle controller. The test result determination module 704 is used to determine the test result corresponding to the test case based on the comparison result of the expected security event log and the actual security event log set.
[0119] For details regarding the specific content involved in the embodiments of this application, please refer to the description of the above method embodiments. For the sake of brevity, these details will not be repeated here.
[0120] Corresponding to the above embodiments, this application also provides a control device.
[0121] See Figure 8 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Figure 8As shown, the control device 800 may include a processor 801, a memory 802, and a communication unit 803. These components communicate via one or more buses. Those skilled in the art will understand that the control device structure shown in the figure does not constitute a limitation on the embodiments of this application. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0122] The communication unit 803 is used to establish a communication channel, thereby enabling the control device to communicate with other devices.
[0123] The processor 801 serves as the control center of the control device, connecting various parts of the control device via various interfaces and lines. It executes software programs and / or modules stored in the memory 802, and calls data stored in the memory to perform various functions of the control device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 801 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.
[0124] Memory 802 is used to store the execution instructions of processor 801. Memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0125] When the execution instructions in memory 802 are executed by processor 801, the control device 800 is able to perform some or all of the steps in the above method embodiments.
[0126] Corresponding to the above embodiments, this application also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program runs, it can control the device where the computer-readable storage medium is located to execute some or all of the steps in the above method embodiments. In specific implementation, the computer-readable storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0127] Corresponding to the above embodiments, this application also provides a computer program product containing executable instructions that, when executed on a computer, cause the computer to perform some or all of the steps in the above method embodiments.
[0128] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0129] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0130] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0131] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0132] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A test method for an on-board controller, characterized in that, include: Obtain test cases corresponding to the intrusion detection system. The test cases include attack vector parameters and expected security event logs corresponding to the attack vector parameters. The intrusion detection system is used to monitor abnormal behavior in the communication of the vehicle controller. According to the attack vector parameters, attack traffic corresponding to the attack vector parameters is injected into the vehicle controller, and the attack traffic is used to characterize the attack behavior against the vehicle controller; During the process of injecting the attack traffic into the vehicle controller, the actual security event log set corresponding to the intrusion detection system is obtained; Based on the comparison between the expected security event log and the actual security event log set, the test result corresponding to the test case is determined.
2. The method according to claim 1, characterized in that, The step of injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters includes: Based on the attack vector parameters, determine the attack injection command corresponding to the attack vector parameters; The attack injection command is sent to the attack injection device, and the attack injection device responds to the attack injection command by injecting attack traffic corresponding to the attack vector parameters into the vehicle controller.
3. The method according to claim 2, characterized in that, The acquisition of the actual security event log set corresponding to the intrusion detection system includes: A log collection command is sent to a log collection device, which responds to the log collection command and collects all actual security event logs corresponding to the intrusion detection system. Receive the actual security event log set sent by the log collection device, which includes all the actual security event logs.
4. The method according to claim 1, characterized in that, The actual security event log set includes the correspondence between actual security event logs and the test cases. Determining the test result corresponding to the test case based on the comparison between the expected security event logs and the actual security event log set includes: Based on the correspondence between actual security event logs and test cases, target actual security event logs corresponding to the test cases are filtered out from the actual security event log set; Based on the comparison between the expected security event log and the actual security event log of the target, the test result corresponding to the test case is determined.
5. The method according to claim 4, characterized in that, The step of determining the test result corresponding to the test case based on the comparison result between the expected security event log and the target actual security event log includes: Based on the comparison results of the expected security event log and the target actual security event log in each of the multiple comparison dimensions, the test result corresponding to the test case is determined.
6. The method according to claim 1, characterized in that, Before obtaining the test cases corresponding to the intrusion detection system, the method further includes: constructing a test case library corresponding to the intrusion detection system based on the functional information of the intrusion detection system; The step of obtaining test cases corresponding to the intrusion detection system includes: obtaining test cases corresponding to the intrusion detection system from the test case library.
7. The method according to claim 6, characterized in that, The step of obtaining test cases corresponding to the intrusion detection system from the test case library includes: obtaining all test cases in the test case library; The step of injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters includes: for each of the test cases in all the test cases, injecting attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters in the test case; The step of determining the test result corresponding to the test case based on the comparison result between the expected security event log and the actual security event log set includes: determining the test result corresponding to each test case based on the comparison result between the expected security event log and the actual security event log set in each test case.
8. The method according to claim 7, characterized in that, The method further includes: Based on the test results corresponding to each test case, a test report corresponding to the intrusion detection system is generated.
9. A testing device for an on-board controller, characterized in that, include: The test case acquisition module is used to acquire test cases corresponding to the intrusion detection system. The test cases include attack vector parameters and expected security event logs corresponding to the attack vector parameters. The intrusion detection system is used to monitor abnormal behavior in the communication of the vehicle controller. An attack traffic injection module is used to inject attack traffic corresponding to the attack vector parameters into the vehicle controller according to the attack vector parameters. The attack traffic is used to characterize the attack behavior against the vehicle controller. The log set acquisition module is used to acquire the actual security event log set corresponding to the intrusion detection system during the process of injecting the attack traffic into the vehicle controller; The test result determination module is used to determine the test result corresponding to the test case based on the comparison result between the expected security event log and the actual security event log set.
10. A control device, characterized in that, include: processor; Memory; And a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, causes the control device to perform the method of any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 8.