Vehicle gateway functional safety and network security horizontal comparison test method and system

Abstract

The application discloses a kind of vehicle gateway functional safety and network security horizontal contrast test method and system, method includes: according to relevant standard construction vehicle gateway standardization contrast test case library, and the input information of test case is configured, using HIL hardware-in-the-loop simulation test, by standard test method determine that function and performance satisfy the vehicle gateway equipment of relevant standard requirement, build vehicle gateway standard test component pool;At least two standard parts are selected from vehicle gateway standard test component pool, standard part and measured piece are configured according to the input information of test case in vehicle gateway standardization contrast test case library, after standard part and measured piece build vehicle gateway contrast test system, test according to the test case in vehicle gateway standardization contrast test case library, output the test result of measured piece.The application can be repeated to obtain more test evaluation results, increase the scientificity and reliability of test result.

Description

Technical Field

[0001] This invention relates to the field of vehicle network security technology, and in particular to a method and system for comparative testing of the functional safety and network security of vehicle gateways. Background Technology

[0002] With the rapid development of automotive intelligence and connectivity technologies, the in-vehicle electronic and electrical architecture is evolving from a distributed to a centralized model. As the core hub of the vehicle network, the in-vehicle gateway undertakes critical functions such as data routing, protocol conversion, and security isolation between different domain controllers (e.g., powertrain domain, chassis domain, infotainment domain). Simultaneously, the in-vehicle gateway also needs to interact with external networks (e.g., V2X, cloud platforms, OTA upgrade servers). Its reliability in functional safety and network security directly affects vehicle driving safety and user privacy protection.

[0003] Existing functional safety and cybersecurity testing methods for vehicle gateways largely rely on manual, one-time testing based on static preset conditions according to relevant standards. Furthermore, due to significant differences in objectives, implementation mechanisms, and testing methods between functional safety and cybersecurity, traditional testing solutions often treat them separately, making it difficult to address collaborative risks in complex scenarios. More importantly, the cybersecurity assessment methods for vehicle gateways are limited, relying primarily on known vulnerability databases. They use discovered vulnerability characteristics to match known vulnerability databases such as CVE or CWE to determine the existence and number of vulnerabilities, failing to effectively detect and discover unknown threats.

[0004] To address the aforementioned issues, there is an urgent need for a systematic comparative analysis testing method, system, and device for the functional safety and network security of vehicle gateways. This device should be able to: 1) perform semi-automatic or fully automated testing of the functional safety and network security of vehicle gateways across multiple dimensions and scenarios; 2) enable coupled testing of functional safety and network security scenarios, with test scenarios that can be flexibly switched and configured autonomously; and 3) promptly and effectively discover unknown threats and vulnerabilities, enriching vulnerability testing and discovery methods and improving test coverage and efficiency. Summary of the Invention

[0005] Technical Objective: To address the shortcomings of existing technologies, this invention discloses a method and system for comparative testing of functional safety and network security of vehicle gateways. It automates the entire comparative testing process, is easy to operate, and allows for repeated testing to obtain more test evaluation results, thereby increasing the scientific validity and credibility of the test results.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution.

[0007] A comparative testing method for functional safety and network security of an in-vehicle gateway includes:

[0008] A standardized comparative test case library for vehicle gateways is constructed based on relevant standards, and the input information for the test cases is configured. The input information includes a CAN network communication matrix and an Ethernet interface control list. The vehicle gateway includes a CAN gateway and an Ethernet gateway. The standardized comparative test case library includes interface access control testing, data frame consistency detection, protocol status detection, message data injection attack detection, message data spoofing attack detection, UDS diagnostic service attack detection, and diagnostic routing detection based on the DoIP protocol.

[0009] Using HIL hardware-in-the-loop simulation testing, standardized testing methods are used to identify vehicle gateway devices that meet the relevant standard requirements in terms of both function and performance. These devices serve as standard test equipment for vehicle gateways, and a pool of standard test components for vehicle gateways is constructed.

[0010] At least two standard components are selected from the standard test component pool of vehicle gateways. The standard components and the device under test (DUT) are configured with the input information of test cases according to the standardized comparison test case library of vehicle gateways. After the standard components and the DUT build the vehicle gateway comparison test system, the test is performed according to the test cases in the standardized comparison test case library of vehicle gateways, and the test results of the DUT are output.

[0011] A comparative testing system for functional safety and network security of an in-vehicle gateway, used to implement the comparative testing method for functional safety and network security of an in-vehicle gateway as described above, includes: a platform operation web front-end and a task response test back-end program;

[0012] The web front-end includes a test target management module, a test case management module, a scenario management module, a task management module, and a report management module;

[0013] The target management module is used to create and modify the relevant name, type, brand, model and device version information of the vehicle gateway devices required for testing. This information is used to select specific test task objects when creating a task, including standard vehicle gateway parts and the device under test.

[0014] The test case management module is used to input and update specific test case information required for testing, including test case name, test basis, test category, comparison test method, network attack method, and the path name of the bound data source configuration file, etc. It enables the task response backend program to call and test different data files and algorithms according to the specific test case parameters when executing specific test case comparison test tasks.

[0015] The scenario management module is used to create different test scenarios corresponding to different function sets of the vehicle gateway. Users can freely create and divide the specific test scenarios faced by the vehicle gateway by independently selecting the specific test cases to be tested, so as to facilitate the use of the scenario when creating test tasks.

[0016] The task management module creates test tasks for specific devices under test in different test scenarios based on the established test scenarios and conducts actual comparative tests.

[0017] The report management module is used to generate comparative test reports for specific devices under test based on the results of specific test tasks. It can identify defects or problems in the functional safety or network security fields of the device under test, or discover coupled unknown defects or problems. At the same time, based on the specific comparative test method of the vehicle gateway, that is, based on the specific functions and performance tested on the vehicle gateway, and combined with the relevant threat intelligence data of the AUTO-ISAC threat classification framework, it analyzes some standardized threat vulnerability information that may be associated with the detected anomalies, and provides relevant suggestions for further reproduction, localization and analysis of the causes of the detected problems.

[0018] The task response backend program includes a task receiving and parsing module, a component scheduling module, a test case scheduling module, an input construction module, a data distribution module, a comparison and adjudication module, an analysis and evaluation module, and a data management module;

[0019] The task receiving and parsing module receives and parses the test task, calls the input construction module to check whether the test environment is ready based on the test component information in the task, and sends the test cases in the test task to the test case management module.

[0020] The component scheduling module is responsible for selecting other functional standard components from the gateway standard test component pool to replace the standard components of the comparative test system if a systemic failure is found in the standard components during the task testing process, such as the absence of standard data output, so as to ensure that the system can operate normally in the next round of comparative testing.

[0021] The test case scheduling module schedules test cases in the task, stores the test result information of individual test cases, refreshes the test case queue according to the test results of the test cases, and adjusts the test cases for the next round using the priority and hierarchy of the test cases to be tested.

[0022] The input module is used to read the relevant configuration parameters of the task device, check whether the test component matches, update the port information of the test component, verify whether the parameters of each test case of the task are correct, and whether the data input configuration file matches. Finally, it constructs the input information required by the test component based on the test case parameters.

[0023] The data distribution module is used to create port data connections between the system and each component under test, inject the constructed input information into each component under test according to the distribution strategy, and parse the output information of the component under test into the data cache in real time for comparison with the test of the adjudication module.

[0024] The comparison and adjudication module calls different adjudication methods based on the test task and specific test case parameter information. This adjudication method performs multi-mode adjudication based on the same dimension data output by the device, and performs single adjudication and iterative adjudication on the standard output information of the standard part and the test part to generate adjudication results.

[0025] The analysis and evaluation module uses the test results data of each test case to perform a simple analysis and statistics on the functional safety and performance safety of the device under test at the end of the overall task test, and outputs a task test report.

[0026] The data management module uses database tables to record the execution results of each test case during the comparison test and the final evaluation results of the task.

[0027] Beneficial effects:

[0028] 1. Compared with the traditional manual testing method for the functional safety and network security of vehicle gateways based on static preset conditions, the present invention realizes full automation of the comparative testing process, is simple to operate, can repeat the test multiple times to obtain more test evaluation results, and increases the scientificity and credibility of the test results.

[0029] 2. The modular configuration adopted in this invention improves the system's flexibility and scalability, and enhances the maintainability of the comparative testing system. It enables flexible configuration and expansion of test equipment, test cases, and test scenarios, significantly shortening the preparation time for equipment testing.

[0030] 3. This invention proposes a horizontal multi-mode adjudication method based on data of the same dimension, which can couple the functional safety and network security testing scenarios of vehicle gateways, and can cope with the integrated testing of functional safety and network security of vehicle gateways in more complex scenarios. Furthermore, unlike traditional vulnerability detection methods based on fault vulnerability knowledge base matching, this invention can not only discover vehicle gateway device faults caused by traditional CVE or CWE vulnerabilities, but also handle the detection of unknown faults or vulnerabilities, thus improving the system's detection capabilities. Attached Figure Description

[0031] Figure 1 This is a hardware topology diagram of a vehicle gateway functional safety and network security comparative testing device according to the present invention.

[0032] Figure 2 This is a block diagram of a comparative testing method for the functional safety and network security of an in-vehicle gateway according to the present invention.

[0033] Figure 3 This is a flowchart of a comparative test method for the functional safety and network security of an in-vehicle gateway according to the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0035] Example 1

[0036] As attached Figure 1 To be continued Figure 3 As shown in this embodiment, a comparative testing method for the functional safety and network security of an in-vehicle gateway includes the following steps:

[0037] Step 1: Construct a standardized comparative test case library for vehicle gateways based on relevant standards, and configure the input information for the test cases. The input information includes a CAN network communication matrix and / or an Ethernet interface control list. The vehicle gateway includes a CAN gateway and an Ethernet gateway. The standardized comparative test case library includes interface access control testing, data frame consistency detection, protocol status detection, message data injection attack detection, message data spoofing attack detection, UDS diagnostic service attack detection, and diagnostic routing detection based on the DoIP protocol.

[0038] Based on relevant standards such as ISO 26262 Functional Safety of Road Vehicles, ISO / SAE 21434 Roadvehicles – Cybersecurity Engineering, and GB / T 40857-2021 Technical Requirements and Test Methods for Information Security of Automotive Gateways, a standardized comparative test case library covering the functional safety and network security of automotive gateways, including CAN gateways and Ethernet gateways, is constructed to address the specific requirements for functional safety, network security, and standardized testing methods. The test cases cover automated comparative test cases for interface access control testing, data frame consistency detection (specific to CAN gateways), protocol state detection (specific to Ethernet gateways), message data injection attack detection, message data spoofing attack detection, UDS diagnostic service attack detection (specific to CAN gateways), and diagnostic routing detection based on the DoIP protocol (specific to Ethernet gateways). Based on the constructed automated comparative test cases, a CAN network communication matrix and / or Ethernet access control list are established. This communication matrix or access control list, as a unified standardized vehicle gateway data forwarding protocol, will be configured on each vehicle gateway device under test and will serve as an important reference for the construction of input data and implementation of specific gateway comparative test cases. If the gateway type of the device under test is a pure CAN gateway, only a CAN network communication matrix needs to be established here; if the gateway type of the device under test is a pure Ethernet gateway, only an Ethernet access control list needs to be established here; if the gateway type of the device under test is a hybrid gateway (generally a hybrid gateway), that is, a combination of CAN gateway and Ethernet gateway, then both the corresponding CAN network communication matrix and Ethernet access control list need to be established simultaneously.

[0039] Among them, the interface access control test refers to sending CAN data frames or Ethernet data packets that conform to the policy to the vehicle gateway data input port specified by the CAN network communication matrix and / or Ethernet interface control list, and detecting whether a corresponding data frame or data payload is received at the data output port specified by the CAN network communication matrix or access control list.

[0040] Data frame conformance testing refers to verifying whether the received CAN data frames conform to the policy specifications based on interface access testing. This includes testing the identifier type, data type, DLC length, period, and signal check value in the CAN data frames.

[0041] Protocol state detection refers to performing state checks on part or all of the TCP / IP session flow, including TCP handshake state, packet length, packet sequence, and TCP session closing state, to verify whether the gateway discards attack packets that do not conform to the policy.

[0042] Message data injection attack detection refers to: constructing high-frequency, high-priority, or high-load-rate message data that conforms to or does not conform to the policy based on the CAN network communication matrix or access control list to simulate a denial-of-service attack and detect the priority control policy of the vehicle gateway for forwarding data.

[0043] Message data spoofing attack detection refers to: constructing message data with modified parameters that do not conform to the policy based on the CAN network communication matrix or access control list, such as modifying the data output port, the identifier type, data type, DLC length, transmission period or signal check value of the CAN frame, or modifying the source IP address, TCP / UDP port number, message priority or message size of the Ethernet access control list, and then sending the spoofed message data to all vehicle gateway devices to check whether the gateway discards the attack data packets that do not conform to the policy;

[0044] UDS diagnostic service attack detection refers to: based on relevant standards, such as ISO 14229 and ISO 15765-2 (CAN) standards, constructing attack CAN message data using the UDS protocol and injecting it into the vehicle gateway under test, and verifying whether the vehicle gateway discards attack CAN messages that do not conform to the policy.

[0045] DoIP-based diagnostic routing detection refers to: when the Ethernet gateway diagnostic function is enabled, identifying and confirming the network connection through IP address information, establishing a TCP connection and activating the gateway routing function, and then transmitting UDS diagnostic data (such as Payload type 0x8001) by encapsulating the DoIP protocol header to perform diagnostic sessions, firmware flashing, real-time monitoring and gateway cross-protocol routing confirmation.

[0046] For the CAN gateway in an onboard gateway, constructing a CAN network communication matrix includes:

[0047] 1) Determine the network topology of the nodes associated with the CAN gateway to obtain the structure of the CAN network;

[0048] 2) Define the relevant message signals in the node network according to the CAN gateway automated comparative test case definition in the vehicle gateway standardized comparative test case library, including signal name and signal parameters. The signal parameters include byte type, start byte, bit length, precision and offset, etc.

[0049] 3) Determine the CAN network transmission message ID, message period, and CAN network routing priority; this step is part of constructing the CAN network communication matrix. The CAN network communication matrix can be understood as a specific gateway CAN data transmission standard. In practice, the CAN gateway device needs to adapt according to the CAN network communication matrix we define.

[0050] 4) Based on the test requirements of the CAN gateway automated comparison test cases, establish ID allocation and routing strategies as the CAN network communication matrix, and convert the CAN communication matrix into an executable and portable DBC file, which contains signal parsing rules and the node network topology relationship of the CAN network.

[0051] For the Ethernet gateway in the vehicle gateway, the Ethernet interface control list (ACL) is constructed as follows:

[0052] 1) Determine the network topology of the nodes associated with the Ethernet gateway to obtain the structure of the Ethernet network;

[0053] 2) Divide VLANs according to the functional domain of the nodes in the Ethernet network, and configure VLAN tags based on VLAN ports to achieve logical isolation;

[0054] 3) Classify the transmitted data protocols to implement data traffic priority settings; for example, time-sensitive protocols (such as AVB / TSN) need to be assigned the highest priority to ensure low-latency transmission, non-real-time data (such as OTA upgrades, diagnostic information) adopt a dynamic bandwidth allocation strategy, and traditional bus protocols (CAN / LIN) need to be converted by the gateway and mapped to the corresponding Ethernet VLAN.

[0055] 4) Define data outbound and inbound rules for each VLAN port to restrict specific types of data traffic to be transmitted on designated ports;

[0056] 5) Finally, perform dynamic IP settings for specific VLAN ports to ensure that critical device IPs are fixed, avoid ACL failure, and configure encrypted data traffic so that encrypted data is only allowed to pass through specified encrypted ports (such as HTTPS 443, IPsec VPN tunnel), blocking unencrypted sensitive data.

[0057] Step 2: Utilize HIL (Hardware-in-the-Loop) simulation testing to identify vehicle gateway devices that meet relevant standard requirements in terms of functionality and performance through standardized testing methods. These devices will serve as standard test equipment for vehicle gateways, and a standard test component pool for vehicle gateways will be constructed. Subsequently, a specific number of standard components will be selected from the standard test component pool for comparative testing with the device under test.

[0058] Step two specifically includes the following:

[0059] Step 2.1: Build a hybrid network including CANoe nodes, Ethernet switches, and load simulators to simulate real-world interaction scenarios of the vehicle gateway; use tools such as CANoe (supports CAN / LIN / Ethernet protocol simulation), Wireshark (SOME / IP / DoIP parsing), Vector toolchain (signal mapping verification), iPerf (Ethernet bandwidth testing), and CANStress (CAN bus load simulation) to build a hybrid network including CANoe nodes, Ethernet switches, and load simulators to simulate real-world interaction scenarios of the vehicle gateway.

[0060] Step 2.2: Perform Protocol Data Unit (PDU) gateway routing control and latency verification. Use CANoe to send CAN / CANFD messages conforming to the routing protocol to verify that CAN data can be correctly forwarded to the Ethernet, conforms to protocol conversion consistency rules, and that the transmission latency is ≤5ms. Send UDP packets via SOME / IP service to verify the data integrity of the CAN bus received frames and the consistency of ID, DLC, and signal values, recording that the transmission latency is ≤10ms. Use CANoe or SOME / IP service to send message data at fixed intervals, testing the data reception stability and timing accuracy at the message data receiving port.

[0061] Step 2.3: Perform security function tests on the vehicle gateway; send broadcast packets between the Ethernet VLAN and CAN subnet to verify the data isolation effect; send Ethernet encrypted diagnostic commands (such as OTA upgrade requests) to verify the integrity of the data after decryption at the CAN end;

[0062] Step 2.4: Perform data throughput and end-to-end latency tests on the vehicle gateway; on the Ethernet side, use iPerf to send a full-load UDP data stream under the 1000Base-T1 interface to verify that the effective throughput is ≥950Mbps; on the CAN side, test the CAN FD interface transmission rate under 90% bus load conditions, requiring an effective throughput ≥4.8Mbps; use CANoe to measure the total end-to-end cross-protocol transmission delay (CAN→Ethernet→CAN), requiring a delay ≤10ms;

[0063] Step 2.5: Conduct stability and stress tests on the vehicle gateway; simulate the simultaneous transmission of four Ethernet video streams (4K 60fps) and two CAN FD control commands (cycle 10ms) for 30 minutes, requiring a frame drop rate ≤0.05%.

[0064] Step 3: Select at least two standard components from the standard test component pool for vehicle gateways, ensuring that the standard components and the device under test (DUT) meet the heterogeneous redundancy requirement. Both the standard components and the DUT are configured with input information for test cases based on the standardized comparative test case library for vehicle gateways. After building the vehicle gateway comparative test system, the standard components and the DUT are tested according to the test cases in the library, and the test results for the DUT are output. The DUT test results are obtained based on the test results of the standard components and the DUT itself. This involves applying a "majority decision" or "majority consensus decision" to the real-time synchronized output information of these heterogeneous actuators, thus outputting the DUT test results.

[0065] The specific adjudication method needs to be applied in accordance with the specific circumstances. For example, "majority consensus adjudication" can be performed for simple state variables, and "continuous consistency" adjudication can be performed for historical data values ​​of variables. In the embodiments of the present invention, a consensus adjudication that combines the above two methods can also be used, or "multi-mode adjudication" that iteratively runs multiple adjudication algorithms can also be used.

[0066] From the standard test component pool for vehicle gateways in Step 2 above, identify at least two standard vehicle gateway components that meet the relevant functional and performance indicators. These standard components and the device under test (DUT) must satisfy heterogeneous redundancy characteristics, determined by the inherent characteristics of different gateway components. Heterogeneity refers to the fact that the gateway devices included in the test component pool come from different manufacturers, or their hardware circuits, underlying software implementations, interface types, routing interaction methods, etc., differ to some extent. This indicates that these gateway components are heterogeneous. Redundancy means that the constructed comparative test system is jointly completed by multiple gateway components that perform the same function. These gateway components are redundant, and if an anomaly is found in the standard component test, other normal gateway components in the component pool can be selected for replacement, providing redundancy backup. Then, perform device conformance verification between the DUT and at least two standard vehicle gateway components. If the DUT device conforms to the verification, configure all standard components and the DUT with a CAN gateway network communication matrix (if a CAN interface is available) and an Ethernet gateway interface control list (ACL, if an Ethernet interface is available). Finally, an automotive gateway comparative testing system was built using no fewer than two standard automotive gateway components and at least one device under test. The system's front end was used to input the device under test's name and model information, the standardized comparative test case library for automotive gateways, and to create relevant test scenarios for specific gateway functions. Then, gateway test tasks were created and tested, and an analysis report containing the comparative test results was output.

[0067] Step three specifically includes the following:

[0068] Step 3.1: Compliance verification of the vehicle gateway under test relative to at least two standard vehicle gateway components. This mainly verifies the differences in interface types (CAN / CANFD interface and Ethernet interface) and quantities of the under test relative to the standard components, in order to determine whether the under test can be configured with the same CAN network communication matrix and Ethernet interface control list (ACL) as the standard components.

[0069] Step 3.2: If the vehicle gateway DUT is verified to be compliant, configure the vehicle gateway DUT with the same CAN network communication matrix and Ethernet interface control list (ACL) as the standard component. This can be achieved through the vehicle gateway's host computer protocol configuration, or by contacting the vehicle gateway manufacturer and providing a pre-designed CAN network communication matrix and Ethernet interface control list (ACL) along with related explanatory documents, allowing the manufacturer to configure it by modifying the protocol firmware.

[0070] Step 3.3: Based on the verification and configuration in Steps 3.1 and 3.2 above, using the minimum functional set of the device under test (DUT) and the standard component as the test content set, connect the DUT and the standard component to the vehicle gateway comparison test system; Step 3.3 includes:

[0071] Step 3.3.1: The hardware connection topology diagram of the vehicle gateway comparison test system is as follows: Figure 1 As shown, the CAN interface of the vehicle gateway is converted into an Ethernet interface through a "CAN-Ethernet converter" and connected to the system server. The Ethernet interface of the vehicle gateway is then connected to the server's Ethernet port through an Ethernet switch.

[0072] Step 3.3.2: The server platform comparison test program, based on its own IP address, the configured IP address of the "CAN-Ethernet converter," the configured IP address of the Ethernet switch, and the corresponding CAN channel port, reads the TCP / UDP data packets converted from the CAN output data protocol of each device, as well as the TCP / UDP data packets directly routed by the gateway Ethernet interface. Simultaneously, the platform program also converts the TCP / UDP data packets to be injected into each device into the CAN interface injection data format via the "CAN-Ethernet converter" or injects them directly into the gateway Ethernet interface.

[0073] Step 3.3.3: The “CAN-Ethernet conversion device” itself acts as a socket connection server and can be configured with a different IP address on the same network segment as the server platform. At the same time, the conversion device and each connected CAN bus can be configured with a unique port number through the conversion device’s host computer as the port for creating a socket connection. In addition, the configured CAN bus baud rate ensures that the conversion device can establish correct data communication with each device’s CAN bus.

[0074] Step 3.4: After the gateway device successfully connects to the system, the front end of the gateway comparison test system needs to input the relevant information of the test target and the data parameters of each specific test case into the system's backend database, and then create specific test case scenarios for the gateway comparison test; Step 3.4 includes:

[0075] Step 3.4.1: Start the front-end and back-end programs of the gateway comparison test system on the server. Enter the name, model, manufacturer and firmware version information of the standard component and the component under test of the gateway through the front-end [Target Management Module] of the gateway comparison test system. If the device information is updated, modify the relevant information through the edit button.

[0076] Step 3.4.2: Enter the relevant information of each comparative test case introduced in Step 1.1 into the front-end [Test Case Management Module] of the gateway comparison test system, including the test case name, comparison test method, network attack method, and the bound test case input data configuration file information;

[0077] Step 3.4.3: By comparing the front end of the gateway test system [Scenario Management Module], establish a set of test cases selected for specific functions of the test vehicle gateway, thereby creating multiple test scenario sets.

[0078] Step 3.5: Through the gateway comparison test system front-end [Task Management Module] - [Create Task] option, select the standard vehicle gateway component and the component under test to be compared and tested, then select the test scenario to be performed, i.e., select the test case set, and finally click the [Test] button to start the vehicle gateway comparison test. During the test, pay attention to the real-time display information of the test case comparison test shown on the system front-end. After the test is completed, go to the front-end [Report Management] module to view the task test results and detailed record information.

[0079] This invention proposes a horizontal multi-mode adjudication method based on data of the same dimension. This method can couple the functional safety and network security testing scenarios of vehicle gateways, enabling integrated testing of functional safety and network security of vehicle gateways in more complex scenarios. Here, "data of the same dimension" refers to the fact that, during horizontal multi-mode adjudication, the same CAN communication matrix or Ethernet forwarding list is used across different vehicle gateway firmware and models. Based on the forwarding protocol, the parsed CAN or Ethernet data is analyzed to include the data to be adjudicated under the same dimension. For example, multi-mode adjudication can be performed on data of a certain dimension, such as CANID, data identifier, data length, data type, message period, and channel load rate, after parsing CAN data from different gateways.

[0080] Unlike traditional vulnerability detection methods based on fault vulnerability knowledge base matching, this invention can not only detect vehicle gateway device faults caused by traditional CVE or CWE vulnerabilities, but also handle unknown faults or vulnerabilities, thus improving the system's detection capabilities.

[0081] This invention also provides a comparative testing system for the functional safety and network security of an in-vehicle gateway, used to implement the comparative testing method for the functional safety and network security of an in-vehicle gateway described above. The system includes: a platform operation web front-end and a task response testing back-end program.

[0082] The platform's web front-end includes modules for test target management, test case management, scenario management, task management, and report management.

[0083] The test target management module is used to create and modify the relevant name, type, brand, model and device version information of the vehicle gateway devices required for testing. This information is used to select specific test task objects when creating a task, including standard vehicle gateway parts and the device under test.

[0084] The test case management module is used to input and update specific test case information required for testing, including test case name, test basis, test category, comparison test method, network attack method, and the path name of the bound data source configuration file, etc. It enables the task response backend program to call and test different data files and algorithms according to the specific test case parameters when executing specific test case comparison test tasks.

[0085] The scenario management module is used to create different test scenarios corresponding to different function sets of the vehicle gateway. Users can freely create and divide the specific test scenarios faced by the vehicle gateway by independently selecting the specific test cases to be tested, so as to facilitate the use of the scenario when creating test tasks.

[0086] The task management module creates test tasks for specific devices under test in different test scenarios based on the established test scenarios and conducts actual comparative tests.

[0087] The report management module generates comparative test reports for specific devices under test (DUTs) based on the results of specific test tasks. It identifies defects or issues in the functional safety or cybersecurity areas of the DUTs, or discovers coupled unknown defects or issues. Simultaneously, based on the specific comparative testing methodology for vehicle gateways—that is, focusing on the specific functions and performance tested on the vehicle gateways—it analyzes relevant threat intelligence data from the AUTO-ISAC threat classification framework to identify standardized threat and vulnerability information that may be associated with detected anomalies. It also provides relevant suggestions for further reproduction, localization, and root cause analysis of the detected problems.

[0088] The task response backend program includes a task receiving and parsing module, a component scheduling module, a test case scheduling module, an input construction module, a data distribution module, a comparison and adjudication module, an analysis and evaluation module, and a data management module.

[0089] The task receiving and parsing module receives and parses the test task, calls the input construction module to check whether the test environment is ready based on the test component information in the task, and sends the test cases in the test task to the test case management module.

[0090] The component scheduling module is responsible for selecting other functional standard components from the gateway standard test component pool to replace the standard components of the comparative test system if a systemic failure is found in the standard components during the task testing process, such as the absence of standard data output, so as to ensure that the system can operate normally in the next round of comparative testing.

[0091] The test case scheduling module schedules test cases in the task, stores the test result information of individual test cases, refreshes the test case queue according to the test results of the test cases, and adjusts the test cases for the next round using the priority and hierarchy of the test cases to be tested.

[0092] The input module is used to read the relevant configuration parameters of the task device, check whether the test component matches, update the port information of the test component, verify whether the parameters of each test case of the task are correct, and whether the data input configuration file matches. Finally, it constructs the input information required by the test component based on the test case parameters.

[0093] The data distribution module is used to create port data connections between the system and each component under test, inject the constructed input information into each component under test according to the distribution strategy, and parse the output information of the component under test into the data cache in real time for comparison with the test of the adjudication module.

[0094] The comparison and adjudication module calls different adjudication methods based on the test task and specific test case parameter information. This adjudication method performs multi-mode adjudication based on the same dimension data output by the device, and performs single adjudication and iterative adjudication on the standard output information of the standard part and the test part to generate adjudication results.

[0095] The analysis and evaluation module uses the test results data of each test case to perform a simple analysis and statistics on the functional safety and performance safety of the device under test at the end of the overall task test, and outputs a task test report.

[0096] The data management module uses database tables to record the execution results of each test case during the comparison test and the final evaluation results of the task.

[0097] This vehicle gateway comparative testing system includes a platform operation web front-end and a task response testing back-end program. The web front-end includes a test target management module, a test case management module, a scenario management module, a task management module, and a report management module. The target management module is used to input information such as the names, types, and models of the standard vehicle gateway components and the device under test (DUT) required for testing; the test case management module is used to input and update specific test case information required for testing; the scenario management module allows users to freely create and divide specific test scenarios for the vehicle gateway by selecting specific test cases; the task management module creates test tasks for specific devices under test in different test scenarios based on the established test scenarios and conducts actual comparative tests; the report management module generates comparative test reports for specific devices under test based on the results of specific test tasks, discovering functional safety or network security defects or problems of the DUT, or discovering unknown coupling defects or problems.

[0098] The task response backend program includes a task receiving and parsing module, a component scheduling module, a test case scheduling module, an input construction module, a data distribution module, a comparison and adjudication module, an analysis and evaluation module, and a data management module. The task receiving and parsing module receives and parses test tasks, calls the input construction module to check if the test environment is ready based on the test component information in the task, and distributes the test cases in the test task to the test case management module. The component scheduling module is responsible for replacing standard components in the comparative test system for systemic failures of the vehicle gateway standard components during task testing, such as the lack of standard data output. The test case scheduling module schedules the test cases in the task and refreshes the test case queue based on the test results, adjusting the next round of test cases based on the priority and hierarchy of the test cases to be tested. The input construction module constructs the input information required by the test components based on the test cases and is responsible for checking the test environment. The data distribution module creates port data connections between the system and each component under test (DUT), injects the constructed input information into each DUT, and parses the DUT output information into a data cache for testing by the comparison and adjudication module. The comparison and adjudication module, based on key parameter information, calls different adjudication methods to perform single and iterative adjudications on the standard outputs of the standard component and the DUT, generating adjudication results. The analysis and evaluation module uses the test results data from each test case to perform functional safety and performance safety analysis and statistics on the DUT, outputting a task test report. The data management module uses database tables to record the execution results of each test case during the comparison test and the final evaluation results of the task.

[0099] This invention first constructs a standardized comparative test case library covering the functional safety and network security of vehicle gateways (including CAN gateways and Ethernet gateways) based on relevant standards (such as ISO 26262, ISO / SAE 21434, and GBT 40857-2021). This library then forms a comprehensive CAN network communication matrix or Ethernet access control list. Next, vehicle gateway devices that meet the relevant standards in terms of function and performance are identified using the standardized testing methods and designated as standard test devices (hereinafter referred to as standard components). Then, the vehicle gateway device under test (hereinafter referred to as the device under test) undergoes conformity verification. Based on its data forwarding interface type and quantity, it is determined whether it can be compared with the standard component in terms of function and performance. If the device conforms, the device under test is configured with the same CAN network communication matrix or Ethernet access control list as the standard component. Next, at least two standard automotive gateway components and at least one component under test will be connected to the established automotive gateway comparison test system. By matching different component models under test, selecting different test case scenarios, and using different comparison test methods, comparative tests will be conducted and relevant test reports will be output.

[0100] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A comparative testing method for functional safety and network security of an in-vehicle gateway, characterized in that, include: A standardized comparative test case library for vehicle gateways is constructed based on relevant standards, and the input information for the test cases is configured. The input information includes a CAN network communication matrix and / or an Ethernet interface control list. The vehicle gateway includes a CAN gateway and an Ethernet gateway. The standardized comparative test case library includes interface access control testing, data frame consistency detection, protocol status detection, message data injection attack detection, message data spoofing attack detection, UDS diagnostic service attack detection, and diagnostic routing detection based on the DoIP protocol. Using HIL hardware-in-the-loop simulation testing, standardized testing methods are used to identify vehicle gateway devices that meet the relevant standard requirements in terms of both function and performance. These devices serve as standard test equipment for vehicle gateways, and a pool of standard test components for vehicle gateways is constructed. At least two standard components are selected from the standard test component pool of vehicle gateways, and the standard components and the device under test (DUT) meet the heterogeneous redundancy characteristics. The standard components and the DUT are configured with the input information of test cases according to the standardized comparison test case library of vehicle gateways. After the standard components and the DUT build the vehicle gateway comparison test system, the test is performed according to the test cases in the standardized comparison test case library of vehicle gateways, and the test results of the DUT are output.

2. The method for comparative testing of functional safety and network security of an in-vehicle gateway according to claim 1, characterized in that, Interface access control testing refers to sending CAN data frames or Ethernet packets that conform to the policy to the data input port of the vehicle gateway specified by the CAN network communication matrix or access control list, and detecting whether a corresponding data frame or data payload is received at the data output port specified by the CAN network communication matrix or access control list. Data frame consistency detection refers to verifying whether the received CAN data frames conform to the policy specifications based on interface access testing. Protocol state detection refers to performing state checks on part or all of the TCP / IP session flow to verify whether the gateway discards attack packets that do not conform to the policy. Message data injection attack detection refers to: constructing high-frequency, high-priority, or high-load-rate message data that conforms to or does not conform to the policy based on the CAN network communication matrix or access control list to simulate a denial-of-service attack and detect the priority control policy of the vehicle gateway for forwarding data. Message data spoofing attack detection refers to: constructing message data with partially modified parameters that does not conform to the policy based on the CAN network communication matrix or access control list, and then sending the spoofed message data to all vehicle gateway devices to check whether the gateway discards the attack data packets that do not conform to the policy. UDS diagnostic service attack detection refers to: constructing attack CAN message data based on relevant standards using the UDS protocol and injecting it into the vehicle gateway under test, and verifying whether the vehicle gateway discards attack CAN messages that do not conform to the policy. Diagnostic route detection based on the DoIP protocol refers to: when the Ethernet gateway diagnostic function is enabled, identifying and confirming the network connection through IP address information, establishing a TCP connection and activating the gateway routing function, and then transmitting UDS diagnostic data by encapsulating the DoIP protocol header to perform diagnostic sessions, firmware flashing, real-time monitoring and gateway cross-protocol route confirmation.

3. The method for comparative testing of functional safety and network security of an in-vehicle gateway according to claim 1, characterized in that: For the CAN gateway in an onboard gateway, constructing a CAN network communication matrix includes: 1) Determine the network topology of the nodes associated with the CAN gateway to obtain the structure of the CAN network; 2) Define the relevant message signals in the node network according to the CAN gateway automated comparison test case definition in the vehicle gateway standardized comparison test case library; 3) Determine the CAN network transmission message ID, message period, and CAN network routing priority; 4) Based on the test requirements of the CAN gateway automated comparison test cases, establish ID allocation and routing strategies as the CAN network communication matrix, and convert the CAN communication matrix into an executable and portable DBC file, which contains signal parsing rules and the node network topology relationship of the CAN network.

4. The method for comparative testing of functional safety and network security of an in-vehicle gateway according to claim 1, characterized in that: The Ethernet interface control list is constructed for the Ethernet gateway in the vehicle gateway and includes: 1) Determine the network topology of the nodes associated with the Ethernet gateway to obtain the structure of the Ethernet network; 2) Divide VLANs according to the functional domain of the nodes in the Ethernet network, and configure VLAN tags based on VLAN ports to achieve logical isolation; 3) Classify the transmitted data protocols to set data traffic priorities; 4) Define data outbound and inbound rules for each VLAN port to restrict specific types of data traffic to be transmitted on designated ports; 5) Configure dynamic IP settings for specific VLAN ports to ensure that critical equipment IPs are fixed, and configure encrypted data traffic to block unencrypted sensitive data.

5. A comparative testing system for the functional safety and network security of an in-vehicle gateway, used to implement the comparative testing method for the functional safety and network security of an in-vehicle gateway as described in any one of claims 1-4, characterized in that, include: The platform operates the web frontend and the task response testing backend program; The web front-end includes a test target management module, a test case management module, a scenario management module, a task management module, and a report management module; The target management module is used to create and modify the relevant name, type, brand, model and device version information of the vehicle gateway devices required for testing. This information is used to select specific test task objects when creating a task, including standard vehicle gateway parts and the device under test. The test case management module is used to input and update specific test case information required for testing, including test case name, test basis, test category, comparison test method, network attack method, and the path name of the bound data source configuration file, etc. It enables the task response backend program to call and test different data files and algorithms according to the specific test case parameters when executing specific test case comparison test tasks. The scenario management module is used to create different test scenarios corresponding to different function sets of the vehicle gateway. Users can freely create and divide the specific test scenarios faced by the vehicle gateway by independently selecting the specific test cases to be tested, so as to facilitate the use of the scenario when creating test tasks. The task management module creates test tasks for specific devices under test in different test scenarios based on the established test scenarios and conducts actual comparative tests. The report management module is used to generate comparative test reports for specific devices under test based on the results of specific test tasks. It can identify defects or problems in the functional safety or network security fields of the device under test, or discover coupled unknown defects or problems. At the same time, based on the specific comparative test method of the vehicle gateway, that is, based on the specific functions and performance tested on the vehicle gateway, and combined with the relevant threat intelligence data of the AUTO-ISAC threat classification framework, it analyzes some standardized threat vulnerability information that may be associated with the detected anomalies, and provides relevant suggestions for further reproduction, localization and analysis of the causes of the detected problems. The task response backend program includes a task receiving and parsing module, a component scheduling module, a test case scheduling module, an input construction module, a data distribution module, a comparison and adjudication module, an analysis and evaluation module, and a data management module; The task receiving and parsing module receives and parses the test task, calls the input construction module to check whether the test environment is ready based on the test component information in the task, and sends the test cases in the test task to the test case management module. The component scheduling module is responsible for selecting other functional standard components from the gateway standard test component pool to replace the standard components of the comparative test system if a systemic failure is found in the standard components during the task testing process, such as the absence of standard data output, so as to ensure that the system can operate normally in the next round of comparative testing. The test case scheduling module schedules test cases in the task, stores the test result information of individual test cases, refreshes the test case queue according to the test results of the test cases, and adjusts the test cases for the next round using the priority and hierarchy of the test cases to be tested. The input module is used to read the relevant configuration parameters of the task device, check whether the test component matches, update the port information of the test component, verify whether the parameters of each test case of the task are correct, and whether the data input configuration file matches. Finally, it constructs the input information required by the test component based on the test case parameters. The data distribution module is used to create port data connections between the system and each component under test, inject the constructed input information into each component under test according to the distribution strategy, and parse the output information of the component under test into the data cache in real time for comparison with the test of the adjudication module. The comparison and adjudication module calls different adjudication methods based on the test task and specific test case parameter information. This adjudication method performs multi-mode adjudication based on the same dimension data output by the device, and performs single adjudication and iterative adjudication on the standard output information of the standard part and the test part to generate adjudication results. The analysis and evaluation module uses the test results data of each test case to perform a simple analysis and statistics on the functional safety and performance safety of the device under test at the end of the overall task test, and outputs a task test report. The data management module uses database tables to record the execution results of each test case during the comparison test and the final evaluation results of the task.

Images