Performance evaluation method and device of electronic control unit and computer equipment

Abstract

The invention relates to a performance evaluation method and device of an electronic control unit and computer equipment. The method comprises the following steps: acquiring ECU (Electronic Control Unit) data of a target electronic control unit under an evaluation type according to an expected evaluation type; determining at least one target test dimension matched with the evaluation type according to a preset test dimension; aiming at each target test dimension, searching target ECU data of the target test dimension corresponding to the evaluation type from the ECU data, and determining a test index value of the target test dimension corresponding to the evaluation type according to the target ECU data; and generating a test result matched with the evaluation type according to the test index value. By adopting the method, the limitation of a single detection rate index can be avoided, so that the accuracy of performance evaluation of the electronic control unit is improved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method, apparatus, and computer device for evaluating the performance of an electronic control unit. Background Technology

[0002] With the development of intelligent connected vehicles, in-vehicle networks (CAN / LIN / Ethernet) face severe cybersecurity threats. Intrusion Detection Systems (IDS), as a core defense measure, are supplied by multiple vendors to different Electronic Control Units (ECUs) to form a layered protection system. These different ECUs can include intelligent driving ECUs, infotainment ECUs, etc.

[0003] As the core of a vehicle's electronic system, the electronic control unit (ECU) is responsible for real-time decision-making and execution of key functions, from engine management and chassis control to intelligent driving assistance. The performance of the ECU directly determines the vehicle's safety and energy efficiency, thus requiring an accurate method for evaluating its performance. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for evaluating electronic control units that can improve the accuracy of performance evaluation of electronic control units, in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for evaluating the performance of an electronic control unit, including:

[0006] According to the desired evaluation type, obtain the ECU data of the target electronic control unit under the evaluation type;

[0007] Based on the preset test dimensions, determine at least one target test dimension that matches the evaluation type;

[0008] For each target test dimension, the target ECU data corresponding to the evaluation type is found from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data.

[0009] Generate test results that match the evaluation type based on at least one of the test metric values.

[0010] In one embodiment, determining at least one target test dimension matching the evaluation type based on a preset test dimension includes:

[0011] When the assessment type is a compliance type, the target testing dimension that matches the assessment type shall include at least one of the consistency analysis dimension, the risk and vulnerability analysis dimension, the coverage analysis dimension, and the weight allocation dimension.

[0012] When the evaluation type is performance type, the target test dimension that matches the evaluation type shall include at least one of the consistency analysis dimension, the risk and vulnerability analysis dimension, the coverage analysis dimension, and the performance indicator dimension.

[0013] In one embodiment, generating a test result matching the evaluation type based on at least one of the test metric values ​​includes:

[0014] Based on at least one of the test indicator values, determine the weight corresponding to each of the at least one test indicator values;

[0015] The evaluation value corresponding to the evaluation type is determined based on each test indicator value and the weight corresponding to each test indicator value.

[0016] The test result matching the evaluation type is determined based on the evaluation value.

[0017] In one embodiment, for each target test dimension, the process involves retrieving target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including:

[0018] When the target test dimension is the consistency analysis dimension, the local data of the target electronic control unit and the associated data of the associated electronic control units are retrieved from the ECU data.

[0019] A difference value is determined based on the local data and the associated data; the difference value is used to characterize the degree of difference between the local data and the associated data.

[0020] The test index value of the target electronic control unit in the consistency analysis dimension is determined based on the difference value.

[0021] In one embodiment, for each target test dimension, the process involves retrieving target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including:

[0022] When the evaluation type is the performance type and the target test dimension is the risk vulnerability analysis dimension, the vulnerability exposure factor and the general vulnerability scoring system score corresponding to the identified vulnerability are determined from the ECU data.

[0023] For each identified vulnerability, a risk score is determined based on the vulnerability exposure factor and the score from the general vulnerability scoring system to characterize the risk level of the identified vulnerability.

[0024] The test index value of the target electronic control unit in the risk vulnerability analysis dimension is determined based on the risk scores of all identified vulnerabilities.

[0025] In one embodiment, for each target test dimension, the process involves retrieving target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including:

[0026] When the target test dimension is the coverage analysis dimension, a first number of completed test targets and a second number of preset test targets are determined from the ECU data; the test targets are the covered attack test scenarios or security functions.

[0027] The test index value of the target electronic control unit in the functional coverage analysis dimension is determined based on the first quantity and the second quantity.

[0028] In one embodiment, for each target test dimension, the process involves retrieving target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including:

[0029] If the assessment type is the compliance type and the target testing dimension is the risk vulnerability analysis dimension, then search for the vulnerability type in the ECU data;

[0030] For each vulnerability type, the risk level and risk score corresponding to that vulnerability type are determined based on the vulnerability type, the preset correspondence between vulnerability type, risk level and risk score.

[0031] Based on the risk level and risk score corresponding to all vulnerability types, determine the target risk score corresponding to the target vulnerability type with the highest risk level.

[0032] The test index value of the target electronic control unit in the risk vulnerability analysis dimension is determined based on the target risk score.

[0033] In one embodiment, for each target test dimension, the process involves retrieving target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including:

[0034] If the assessment type is the compliance type and the target test dimension is the weight allocation dimension, determine the type identifier of the target electronic control unit from the ECU data;

[0035] The allocation weight corresponding to the type identifier is determined based on the correspondence between the type identifier and the preset type identifier and allocation weight;

[0036] The test index value of the target electronic control unit in the weight allocation dimension is determined based on the allocated weights.

[0037] Secondly, this application also provides a performance evaluation device for an electronic control unit, comprising:

[0038] The data acquisition module is used to acquire ECU data of the target electronic control unit under the desired evaluation type.

[0039] The test dimension determination module is used to determine at least one target test dimension that matches the evaluation type based on preset test dimensions.

[0040] The testing module is used to, for each target testing dimension, search the ECU data for the target testing dimension corresponding to the evaluation type from the ECU data, and determine the test index value of the target testing dimension corresponding to the evaluation type based on the target ECU data;

[0041] An evaluation module is used to generate test results that match the evaluation type based on at least one of the test indicator values.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0043] According to the desired evaluation type, obtain the ECU data of the target electronic control unit under the evaluation type;

[0044] Based on the preset test dimensions, determine at least one target test dimension that matches the evaluation type;

[0045] For each target test dimension, the target ECU data corresponding to the evaluation type is found from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data.

[0046] Generate test results that match the evaluation type based on at least one of the test metric values.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0048] According to the desired evaluation type, obtain the ECU data of the target electronic control unit under the evaluation type;

[0049] Based on the preset test dimensions, determine at least one target test dimension that matches the evaluation type;

[0050] For each target test dimension, the target ECU data corresponding to the evaluation type is found from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data.

[0051] Generate test results that match the evaluation type based on at least one of the test metric values.

[0052] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0053] According to the desired evaluation type, obtain the ECU data of the target electronic control unit under the evaluation type;

[0054] Based on the preset test dimensions, determine at least one target test dimension that matches the evaluation type;

[0055] For each target test dimension, the target ECU data corresponding to the evaluation type is found from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data.

[0056] Generate test results that match the evaluation type based on at least one of the test metric values.

[0057] The aforementioned performance evaluation method, system, computer equipment, computer-readable storage medium, and computer program product for electronic control units (ECUs) acquire ECU data of the target ECU corresponding to the evaluation type, obtain at least one target test dimension matching the evaluation type from preset test dimensions, determine the corresponding target ECU data from the ECU data for each target test dimension, determine the test index value of the target test dimension based on the target ECU data, and generate test results matching the evaluation type based on multiple test index values. This approach evaluates the performance of ECUs based on multiple target test dimensions determined from consistency analysis, risk and vulnerability analysis, coverage analysis, weight allocation, and performance index dimensions. The evaluation covers a wide range and can be directly utilized when comparing different suppliers later. Compared to evaluation based on a single test dimension, the introduction of multi-dimensional test indicators and the collaborative analysis of different test dimensions generate a comprehensive evaluation result, avoiding the limitations of a single detection rate indicator and thus improving the accuracy of ECU performance evaluation. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is an application environment diagram of the performance evaluation method for an electronic control unit in one embodiment;

[0060] Figure 2 This is a flowchart illustrating a performance evaluation method for an electronic control unit in one embodiment;

[0061] Figure 3 This is a flowchart illustrating a method for determining test index values ​​in one embodiment;

[0062] Figure 4 This is a flowchart illustrating a method for determining test metric values ​​for the risk vulnerability analysis dimension in one embodiment.

[0063] Figure 5 This is a flowchart illustrating a method for determining test metric values ​​for the risk vulnerability analysis dimension in one embodiment.

[0064] Figure 6 This is a flowchart illustrating the performance evaluation method for an electronic control unit in another embodiment;

[0065] Figure 7This is a flowchart illustrating the performance evaluation method for the electronic control unit in yet another embodiment;

[0066] Figure 8 This is a structural block diagram of a performance evaluation device for an electronic control unit in one embodiment;

[0067] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] The performance evaluation method for electronic control units provided in this application can be applied to, for example... Figure 1 In the application environment shown, vehicle terminal 102 communicates with computer device 104 via a network. A data storage system can store the data that computer device 104 needs to process. The data storage system can be integrated onto computer device 104, or it can be located in the cloud or on other network servers. Vehicle terminal 102 can be, but is not limited to, various types of vehicles. Computer device 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0070] The vehicle-side unit 102 uploads test data from different test dimensions of the ECU to the computer device 104. The test data includes at least the ECU data of the target electronic control unit, which in turn includes test data and corresponding security policies for network layer attacks, application layer attacks, and vulnerability verification. Network layer attacks, application layer attacks, and vulnerability verification can be implemented, but are not limited to, existing methods, and will not be elaborated upon here. The computer device 104 acquires the ECU data of the target electronic control unit under the desired evaluation type; it determines multiple target test dimensions matching the evaluation type based on preset test dimensions; these preset test dimensions include consistency analysis, risk and vulnerability analysis, coverage analysis, weight allocation, and performance index dimensions; for each target test dimension, it searches the ECU data for the target test dimension corresponding to the evaluation type, and determines the test index value corresponding to the evaluation type based on the target ECU data; finally, it generates test results matching the evaluation type based on multiple test index values.

[0071] In one exemplary embodiment, such as Figure 2As shown, a performance evaluation method for an electronic control unit is provided, which can be applied to... Figure 1 Taking a server as an example, the process can be a cloud server, including steps 202 to 208. Wherein:

[0072] Step 202: Obtain the ECU data of the target electronic control unit under the desired evaluation type.

[0073] The electronic control unit (ECU) can be an embedded computing device in a vehicle used to control specific functions, enabling control and data interaction between various vehicle subsystems. The ECU communicates with other ECUs via the vehicle network to execute control logic. For example, an ECU may include, but is not limited to, one or more of the following: intelligent driving ECU, infotainment ECU, and body control ECU. There can be one or more target ECUs; this example illustrates the concept using each target ECU as an example.

[0074] The desired assessment type can be the type selected by the user based on testing needs. Furthermore, this operation can be achieved through classification based on vehicle safety level, testing objectives, or regulatory requirements, thus providing a basis for subsequent test dimension selection. Assessment types include at least compliance and performance types. The compliance type can be vehicle safety compliance, and compliance can be, but is not limited to, R155 compliance. The performance type can be, but is not limited to, ECU attack scenario scoring. ECU data can be uploaded from the vehicle to a cloud server; the storage method and format of ECU data on the cloud server can be implemented using existing methods and will not be elaborated here. ECU data can be the raw data set generated by the electronic control unit during operation, which can be used as the data foundation for performance evaluation, reflecting the ECU's operating status under different assessment types.

[0075] The ECU data varies depending on the assessment type. For compliance-related assessments, the ECU data can include ECU test data determined under different testing dimensions. This data can include at least ECU attack log data, vulnerability types, tested security functions, and type identifiers. For performance-related assessments, the ECU data can include at least the electronic control unit's security policy, vulnerability exposure factors and scores from a general vulnerability scoring system for identified vulnerabilities, the number of covered attack test scenarios and the number of preset covered attack test scenarios, as well as the recall and false positive rates of attack tests.

[0076] For example, acquire relevant ECU data such as the ECU name, ECU supplier name, ECU activation method, and the openness of ECU-related interface testing processes (e.g., UTAG port, JTAG port). Based on this data, electronic control unit performance evaluation can be performed. The data is organized, and scores are assigned based on test case feedback results, test case pass rate, and test case testing perspective. For instance, selecting the same type of gateway from different suppliers, executing the same set of test cases, calculating the test case pass rate and the depth of test perspective coverage for each supplier, and then comparing the performance / quality of ECUs from different manufacturers. Step 204: Based on preset test dimensions, determine at least one target test dimension that matches the evaluation type.

[0077] The preset test dimensions can be multiple pre-defined analytical dimensions used to evaluate the performance of electronic control units. For example, preset test dimensions can be, but are not limited to, those set based on security standards, industry specifications, or historical experience. Preset test dimensions can include, but are not limited to, one or more of the following: consistency analysis dimensions, risk and vulnerability analysis dimensions, coverage analysis dimensions, weight allocation dimensions, and performance indicator dimensions. The consistency analysis dimension can determine whether the data or behavior between the target electronic control unit and other related electronic control units is consistent. The consistency analysis dimension can work in conjunction with the risk and vulnerability analysis dimension to improve the accuracy of anomaly identification.

[0078] Consistency analysis testing can include consistency checks between ECUs, which can be achieved by determining the consistency of detection results among ECUs under the same attack scenario. It's important to note that these ECUs refer to ECUs with the same functionality from different manufacturers. For example, consistency checks between ECUs can be the consistency between the actual performance data and the expected performance data of ECUs from different manufacturers facing an attack under the same attack scenario. Actual performance data can be, but is not limited to, attack success rate, attack detection and response time, recall rate, etc.

[0079] Consistency analysis can also be a policy consistency analysis dimension. Testing the policy consistency analysis dimension can involve assessing the similarity between security policies across different ECUs. The testing for risk vulnerability analysis dimensions differs depending on the assessment type. For compliance-type risk vulnerability analysis dimensions, testing involves mapping detected vulnerability types to quantified risk values ​​based on a standardized vulnerability knowledge base (CWE_RISK_MAP). For performance-type risk vulnerability analysis dimensions, testing involves multiplying the vulnerability exposure factor of identified vulnerabilities by the score from a general vulnerability scoring system, and then summing the products.

[0080] Coverage analysis dimensions can include functional coverage analysis and scenario coverage analysis. Scenario coverage analysis testing refers to determining scenario coverage based on the number of covered attack test scenarios and the preset number of covered attack test scenarios. Functional coverage analysis testing determines functional coverage based on the list of tested security functions and the baseline security function list. Weight allocation testing can determine the corresponding weights based on the type of ECU. Performance metric testing can be the recall rate determined by the number of detected real attacks / total number of real attacks, and the false positive rate determined by the number of false positives / total number of normal events.

[0081] Understandably, retrieving multiple target test dimensions that match the evaluation type from preset test dimensions can be achieved by filtering relevant test dimensions based on the evaluation type. This operation can be accomplished by retrieving matching dimensions through configuration files or database queries, thereby ensuring that the test dimensions are consistent with the evaluation target.

[0082] Furthermore, in an exemplary embodiment, when the evaluation type is compliance type, the target test dimension matching the evaluation type is determined to include at least one of the following: consistency analysis dimension, risk vulnerability analysis dimension, coverage analysis dimension, and weight allocation dimension; when the evaluation type is performance type, the target test dimension matching the evaluation type is determined to include at least one of the following: consistency analysis dimension, risk vulnerability analysis dimension, coverage analysis dimension, and performance indicator dimension.

[0083] Step 206: For each target test dimension, find the target ECU data corresponding to the evaluation type from the ECU data, and determine the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data.

[0084] The test index values ​​can be numerical values ​​reflecting the specific performance of the electronic control unit (ECU) under a specific test dimension, and can be used as the basis for generating test results. In one specific embodiment, the test index values ​​are quantitatively analyzed based on ECU data under the target test dimension. Furthermore, the test index values ​​may include, but are not limited to, one or more of the following: ECU consistency, strategy consistency, functional coverage, scenario coverage, recall rate, false positive rate, and vulnerability risk value.

[0085] For each target test dimension, determining the corresponding target ECU data from the ECU data can be achieved by extracting a subset of data relevant to the current test dimension from the ECU data. This subset of data can be obtained through methods such as data filtering and feature extraction, thereby providing accurate data support for generating test metric values. Here, no specific limitations are placed on the method of determining the data subset.

[0086] Determining test index values ​​for target test dimensions based on target ECU data can be achieved by analyzing the target ECU data and generating quantitative indicators. The analysis methods can include, but are not limited to, statistical analysis, machine learning models, and other methods to generate indicator values, thereby forming evaluable numerical results.

[0087] For example, taking compliance as the assessment type, the multiple target test dimensions include at least one of consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension. For each target test dimension, the corresponding target ECU data is determined from the ECU data, and the test index value of the target test dimension is determined based on the target ECU data.

[0088] Step 208: Generate test results that match the evaluation type based on at least one test indicator value.

[0089] The test results can be the final output of the performance evaluation of the electronic control unit (ECU), and can be used to provide an evaluation conclusion on the overall performance of the ECU. For example, the test results can be weighted or comprehensively judged based on multiple test index values. Furthermore, the test results can include evaluation results or comprehensive scoring results determined by at least two dimensions.

[0090] Generating test results that match the evaluation type based on multiple test indicator values ​​can be achieved by integrating multiple test indicator values ​​and outputting a final evaluation result. For example, this operation can be achieved by generating a comprehensive result through weighted averaging, decision tree models, etc., thereby improving the accuracy of electronic control unit performance evaluation.

[0091] The aforementioned performance evaluation method for electronic control units (ECUs) determines the evaluation type of the ECU, acquires the ECU data of the target ECU corresponding to the evaluation type, obtains multiple target test dimensions matching the evaluation type from preset test dimensions, identifies the corresponding target ECU data for each target test dimension from the ECU data, determines the test index value for the target test dimension based on the target ECU data, and generates test results matching the evaluation type based on multiple test index values. By introducing multi-dimensional test indicators and combining the collaborative analysis of different test dimensions, a comprehensive evaluation result is generated, avoiding the limitations of a single detection rate indicator and thus improving the accuracy of ECU performance evaluation. Furthermore, it allows for the selection of corresponding target test dimensions according to different needs, supporting the requirements of different application scenarios.

[0092] The following provides a performance evaluation method for electronic control units with an evaluation type of compliance and multiple target testing dimensions, including at least two of the following: consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension.

[0093] In one exemplary embodiment, such as Figure 3 As shown, a method for determining test index values ​​is provided, including the following steps:

[0094] Step 302: When the target test dimension is the consistency analysis dimension, search for the local data of the target electronic control unit and the associated data of the associated electronic control units in the ECU data.

[0095] The local data acquired varies depending on the assessment type. When the assessment type is compliance-based, the local data includes local ECU attack log data, and the corresponding associated data includes associated ECU attack log data of the associated electronic control unit. When the assessment type is performance-based, the local data includes the first security policy, and the corresponding associated data includes the second security policy of the associated electronic control unit.

[0096] Furthermore, the ECU attack log data referred to in the terms "local ECU attack log data" and "related ECU attack log data" can be understood as the actual performance data of the target electronic control unit in response to an attack. Related ECU attack log data can be the performance data of associated ECUs under the same attack as the target ECU, where the target ECU and associated ECUs have the same function, differing only in that they come from different manufacturers. ECU attack log data can be a measure of the "performance" of the IDS on the ECU, such as recall rate, detection rate, etc. The security policies referred to in the first and second security policies can be, but are not limited to, the original security policies, for example, including but not limited to CAN Flooding rules, UDS unauthorized access rules, alarm level thresholds, etc. Feature decomposition can be achieved through natural language processing, policy modeling, or rule parsing.

[0097] Step 304: Determine the difference value based on the local data and the associated data; the difference value is used to characterize the degree of difference between the local data and the associated data.

[0098] The determination of the difference value can employ statistical analysis methods (such as standard deviation and hypothesis testing), distance metrics (such as Euclidean distance and cosine similarity), information theory metrics (such as KL divergence and cross-entropy), similarity coefficients (such as Pearson correlation coefficient and Jaccard index), distribution comparison methods (such as KS test), and domain-specific indicators (such as edit distance and PSNR in images). Specific methods can be implemented using existing technologies and will not be elaborated upon here. The difference value can be a quantitative indicator characterizing the degree of difference between the local data of the target ECU and the associated data of the related ECUs, and can be used as an intermediate evaluation indicator in the consistency analysis dimension.

[0099] Step 306: Determine the test index value of the target electronic control unit in the consistency analysis dimension based on the difference value.

[0100] Determining the test index value of the target electronic control unit in the consistency analysis dimension based on the difference value can be achieved by mapping the difference value to a preset test index value range to generate test index values ​​under the consistency dimension. For example, it can be achieved, but is not limited to, converting the difference value into test index value using a linear mapping method, or using a non-linear function (such as Sigmoid) to map the index value.

[0101] For consistency analysis dimensions, the following two scenarios apply to different evaluation types:

[0102] Scenario 1: If the assessment type is compliance type and the target test dimension is consistency analysis dimension, then determine the ECU attack log data from the ECU data, and obtain the associated ECU attack log data related to the target electronic control unit; based on the ECU attack log data and the associated ECU attack log data, determine the test index value of the target electronic control unit in the consistency analysis dimension.

[0103] ECU attack log data can be understood as the actual performance data of the target electronic control unit in response to an attack. Associated ECU attack log data can be the performance data of associated ECUs under the same attack as the target ECU. The target ECU and associated ECUs have the same function, differing only in that they come from different manufacturers. ECU attack log data can be used as a metric to measure the "performance" of the IDS on the ECU, such as recall rate, detection rate, etc.

[0104] Taking ECU attack log data as an example of ECU recall rate, we calculate the average of all ECU recall rates and the average of the squares of the deviations of each ECU recall rate from the mean, and obtain the corresponding variance. We then take the square root of the variance to obtain the corresponding dispersion. We then map the dispersion inversely to a consistency index. Based on this consistency index, we determine the consistency of the ECU detection rate, which gives us the test index value of the target electronic control unit in the consistency analysis dimension.

[0105] Furthermore, the consistency analysis dimension can also be based on the above methods to perform consistency analysis on all ECUs of the entire vehicle, obtaining the detection stability of different ECUs under the same attack scenario. It is understandable that in an excellent, well-coordinated vehicle IDS protection system, the various components (IDS on different ECUs) should perform similarly when facing the same batch of attacks, rather than some performing extremely well and others extremely poorly. This "similarity in performance" is consistency.

[0106] In the above embodiments, by using the discreteness of ECU attack log data and associated ECU attack log data to reflect consistency, the consistency between ECUs can be determined intuitively and accurately, and resource consumption can be reduced through simple calculations.

[0107] Scenario 2: If the evaluation type is performance-based and the target test dimension is strategy consistency analysis, then determine the first safety strategy from the ECU data and obtain the second safety strategies for other electronic control units besides the target electronic control unit; perform feature decomposition on the first safety strategy and each second safety strategy to obtain the first feature vector corresponding to the first safety strategy and the second feature vector corresponding to each second safety strategy; calculate the similarity between the first feature vector and each second feature vector to determine the test index value of the target electronic control unit in the consistency analysis dimension.

[0108] The security policy here can be, but is not limited to, the original security policy. For example, it can include, but is not limited to, CAN Flooding rules, UDS illegal access rules, alarm level thresholds, etc. Feature decomposition can be implemented through natural language processing, policy modeling, or rule parsing. For example, feature decomposition can be performed on the first security policy and each of the second security policies to determine whether features such as CAN Flooding rules, UDS illegal access rules, and alarm level thresholds exist. Each feature corresponds to a dimension in the vector, and its value can be binary (0 / 1 represents presence or absence) or numerical (such as the threshold size).

[0109] Similarity can be a numerical metric used to measure the degree of matching between two feature vectors. It can be calculated using methods such as cosine similarity, Euclidean distance, and Jaccard coefficient. For example, cosine similarity can be used to calculate the pairwise cosine similarity between the first feature vector and each of the second feature vectors, and the average value can be taken as the overall policy consistency score. For example, the closer the policy consistency score is to 1, the more consistent the directions of the two vectors are, that is, the more similar their policy logic is.

[0110] For example, policy consistency analysis can be calculated using the method cosine_similarity(ECU_policies), where cosine_similarity represents cosine similarity and ECU_policies represents the original security policies.

[0111] For performance-type assessments where the target test dimension is policy consistency analysis, this method identifies the policy consistency analysis dimension within the performance-type assessment. It extracts the first security policy from the target electronic control unit (ECU) and obtains the second security policy from other ECUs. The two policies are then feature-decomposed to generate feature vectors, and the similarity between these feature vectors is calculated as a test metric. This feature extraction and similarity calculation enable a quantitative expression of policy consistency, allowing for a systematic assessment of the security policy synergy between ECUs and improving the accuracy of performance evaluation.

[0112] In the above embodiments, when the target test dimension is the consistency analysis dimension, the local data of the target electronic control unit and the associated data of the associated electronic control units are searched from the ECU data; the difference value is determined based on the local data and the associated data; and the test index value of the target electronic control unit in the consistency analysis dimension is determined based on the difference value. By introducing the consistency analysis dimension, the local data of the target ECU and the associated data of the associated ECU are compared, and the difference value is calculated, thereby quantitatively evaluating the data consistency performance of the target ECU in the process of interacting with other ECUs. This test index value construction method based on data consistency analysis can effectively identify potential performance problems caused by data tampering, avoid the dependence on single-point data in traditional evaluation methods, and improve the comprehensiveness and accuracy of the evaluation.

[0113] In one exemplary embodiment, such as Figure 4 As shown, a method for determining the test metric value of the risk vulnerability analysis dimension is provided, including the following steps:

[0114] Step 402: When the evaluation type is performance type and the target test dimension is risk vulnerability analysis dimension, determine the vulnerability exposure factor and the general vulnerability scoring system score corresponding to the identified vulnerability from the ECU data.

[0115] Step 404: For each identified vulnerability, determine a risk score based on the vulnerability exposure factor and the score from the general vulnerability scoring system to characterize the risk level of the identified vulnerability.

[0116] Step 406: Determine the test index value of the target electronic control unit in the risk vulnerability analysis dimension based on the risk scores of all identified vulnerabilities.

[0117] The vulnerability exposure factor is not a standard CVSS value, but a dynamically adjusted coefficient based on the contextual information of the identified vulnerability in a specific vehicle environment; for example, it can be between 0 and 1. In one exemplary embodiment, the vulnerability exposure factor can be determined based on factors such as the vulnerability's location in the system, access path, and communication interface. In another exemplary embodiment, the vulnerability exposure factor can be determined based on assessing whether the ECU containing the vulnerability is easily accessible (e.g., a higher exposure factor on a T-Box than on an internal CAN node), and / or whether physical access or high-level privileges are required; and / or whether existing security controls (such as firewall rules or IDS signatures) can reduce the probability of the vulnerability being exploited. The general vulnerability scoring system score can be a standardized score used to characterize the severity of a vulnerability, reflecting its potential harm.

[0118] The first parameter value can be the product of the vulnerability exposure factor of an identified vulnerability and the score of a general vulnerability scoring system, and can be used to characterize the overall risk level of a single vulnerability in a specific ECU. The test index value of the target electronic control unit in the risk vulnerability analysis dimension can be obtained by summing the first parameter values ​​of all identified vulnerabilities.

[0119] For example, when the evaluation type is performance-based and the target test dimension is risk and vulnerability analysis, the vulnerability exposure factor and CVSS score of the identified vulnerabilities are extracted from the ECU data. These are multiplied to obtain the first parameter value for each vulnerability. The sum of the first parameter values ​​for all vulnerabilities is then used as the test metric value for the ECU in that test dimension. The test metric value for the ECU in that test dimension can be calculated using the formula ∑(CVSS_score × exposure_factor), where CVSS_score represents the general vulnerability scoring system score, and exposure_factor represents the vulnerability exposure factor.

[0120] In the above embodiments, the vulnerability exposure factor and CVSS score of the identified vulnerabilities are extracted from the ECU data, multiplied by each to obtain the first parameter value of each vulnerability, and the first parameter values ​​of all vulnerabilities are summed to serve as the test index value of the ECU in this test dimension. By combining the general score of the vulnerability with the degree of exposure in a specific ECU, a test index value with context-aware capability is formed, ensuring the accuracy of the performance evaluation of the electronic control unit.

[0121] In another exemplary embodiment, such as Figure 5 As shown, a method for determining the test metric value of the risk vulnerability analysis dimension is provided, specifically including:

[0122] Step 502: If the assessment type is compliance type and the target testing dimension is risk and vulnerability analysis dimension, find the vulnerability type from the ECU data.

[0123] Step 504: For each vulnerability type, determine the risk level and risk score corresponding to that vulnerability type based on the corresponding relationship between the vulnerability type, the preset vulnerability type, risk level, and risk score.

[0124] Step 506: Based on the risk level and risk score corresponding to all vulnerability types, determine the target risk score corresponding to the target vulnerability type with the highest risk level.

[0125] Step 508: Determine the test index values ​​of the target electronic control unit in the risk vulnerability analysis dimension based on the target risk score.

[0126] The correspondence between vulnerability types, preset vulnerability types, risk levels, and risk scores can be determined based on a standardized vulnerability knowledge base. This standardized knowledge base can be a dataset storing predefined vulnerability attributes and their associated risk levels, used to support matching queries between vulnerability types and quantified risk indices. For example, a standardized vulnerability knowledge base can be built by integrating internationally recognized vulnerability scoring systems (such as CVE / CVSS / CWE_RISK_MAP). Vulnerability types can include, but are not limited to, existing vulnerability types. For instance, the CWE risk level mapping table includes vulnerability types and their corresponding risk indices, as shown in Table 1.

[0127] Table 1

[0128]

[0129] For example, the specific vulnerability type of the target ECU is extracted from the ECU data. This vulnerability type is then compared with a standardized vulnerability knowledge base to obtain a predefined risk level and risk score associated with it. Based on the risk level and risk score, the risk score is quantified to generate a test index value for the ECU in the risk vulnerability analysis dimension. For instance, the ratio of the risk score to a preset risk score value can be determined, and the difference between 1 and the preset risk score ratio is used to obtain the test index value for the ECU in the risk vulnerability analysis dimension. Further, if multiple vulnerability types exist, the maximum risk score is selected as the representative of the overall risk. Finally, the ratio of the maximum risk score to the preset risk score value is calculated, and the difference between 1 and the preset risk score ratio is used to obtain the test index value for the ECU in the risk vulnerability analysis dimension. The score of the test index value in this risk vulnerability analysis dimension can be between 0 and 1, with a lower value indicating a higher vulnerability risk in the system. Furthermore, if a vulnerability exists, or if it doesn't exist but the preset vulnerability type doesn't include that vulnerability, then the risk score is set to the preset default value, for example, 5. This can be determined according to the calculation method of the test index value described above, which will not be repeated here. Taking the compliance and security assessment of the vehicle communication module as an example, the performance evaluation method of the electronic control unit in this embodiment can be as follows: when conducting a compliance type assessment of the infotainment ECU, if its vulnerability type is determined to be "unauthorized access vulnerability", a standardized vulnerability knowledge base is called to find that the risk index corresponding to this type of vulnerability is 8.1 (high risk level). This risk index is directly used as a test index value to characterize the security weakness of the ECU under the risk vulnerability analysis dimension and is included in the final compliance assessment report.

[0130] In the above embodiments, the vulnerability type is determined from the ECU data, and a risk index that maps to the vulnerability type is determined based on a standardized vulnerability knowledge base. This results in the test index value of the target electronic control unit in the risk vulnerability analysis dimension, which can ensure the objectivity and consistency of the data judgment.

[0131] In an exemplary embodiment, for each target test dimension, the target ECU data corresponding to the evaluation type for the target test dimension is retrieved from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data, including:

[0132] When the target test dimension is the coverage analysis dimension, the first number of completed test targets and the second number of preset test targets are determined from the ECU data; the test targets are the covered attack test scenarios or security functions; the test index values ​​of the target electronic control unit in the functional coverage analysis dimension are determined based on the first number and the second number.

[0133] The coverage analysis dimension can be a dimension used to evaluate the completeness of the test data's coverage of the electronic control unit's functions and a dimension for evaluating the completeness of the attack test scenarios. The attack test scenarios can be a set of test cases simulating actual attack behaviors, used to verify the security of the electronic control unit against cyberattacks. Attack test scenarios can include, but are not limited to, one or more of data anomaly testing, data duplication testing, and service anomaly testing. Security functions can be functional modules used to ensure the safe operation of the electronic control unit, ensuring the vehicle has the necessary security response capabilities in the face of abnormal situations. Security functions can include, but are not limited to, one or more of intrusion detection functions, communication encryption functions, and access control functions. The test index value of the target electronic control unit in the functional coverage analysis dimension is determined based on the first quantity and the second quantity, which can be calculated by the ratio of the first quantity to the second quantity.

[0134] For target testing dimensions that are coverage analysis dimensions, there are two scenarios:

[0135] Scenario 1: In an exemplary embodiment, implementation steps for evaluating the functional coverage analysis dimensions of the assessment type are provided, specifically including:

[0136] If the assessment type is compliance and the target test dimension is functional coverage analysis, then determine the list of tested safety functions from the ECU data; determine the ratio of the number of safety functions in the list of tested safety functions to the number of safety functions in the preset baseline safety function list, and obtain the test index value of the target electronic control unit in the functional coverage analysis dimension.

[0137] The tested security function list refers to the set of security functions actually extracted from the target ECU through automated tools or manual auditing, reflecting the security protection capabilities currently implemented by the ECU. For example, it may include, but is not limited to, one or more of remote authentication, firmware signature verification, and intrusion detection response. The baseline security function list can be a set of security functions developed according to regulations, standards, or internal enterprise requirements, representing the minimum security requirements that the ECU must meet. For example, it may include, but is not limited to, one or more of security functions such as access control, secure boot, secure communication, event auditing, and threat detection.

[0138] For example, under the condition that the assessment type is compliance and the target test dimension is functional coverage analysis, data reflecting the execution of safety function tests is extracted from the ECU data to form a list of tested safety functions. This list is then compared with a preset benchmark safety function list to calculate the coverage ratio in terms of the number of functions. Further, this operation can extract the list of tested safety functions by parsing the ECU's test logs or diagnostic response information, and use set operations to calculate the intersection ratio between the tested safety functions and the benchmark safety functions. This ratio is used as a quantitative indicator of functional coverage, thereby achieving a quantitative assessment of the electronic control unit's safety function coverage completeness and ensuring that it meets the compliance requirements regarding the scope of protection capabilities.

[0139] In the above embodiments, a list of tested safety functions is extracted from the ECU data, and the ratio of the number of functions is calculated between the list and a preset benchmark safety function list to generate corresponding test index values. By transforming the degree of function coverage from a qualitative judgment to a quantitative expression, the objectivity and comparability of the evaluation results are enhanced, and the accuracy of electronic control unit performance evaluation is improved.

[0140] Scenario 2: In an exemplary embodiment, implementation steps for scenario coverage analysis dimensions of performance types are provided. Specifically, for each target test dimension, the corresponding target ECU data is determined from the ECU data, and the test index value of the target test dimension is determined based on the target ECU data, including:

[0141] If the evaluation type is performance-based and the target test dimension is scenario coverage analysis, then the number of covered attack test scenarios and the number of preset covered attack test scenarios are determined from the ECU data; based on the ratio of the number of covered attack test scenarios to the number of preset covered attack test scenarios, the test index value of the target electronic control unit in the functional coverage analysis dimension is determined.

[0142] The number of covered attack test scenarios can be a statistical value reflecting the number of attack test scenarios actually executed and covered by the electronic control unit during testing. For example, the number of covered attack test scenarios may include, but is not limited to, one or more of the following: the number of executed denial-of-service attack test cases, the number of injection attack test cases, and the number of communication tampering test cases. The preset number of covered attack test scenarios can be the total number of attack test scenarios that should be covered in advance. The ratio of the number of covered attack test scenarios to the preset number of covered attack test scenarios can be calculated using the method covered_scenarios / total_scenarios, where covered_scenarios represents the number of covered attack test scenarios, and the number of covered attack test scenarios represents the preset number of covered attack test scenarios.

[0143] For example, a coverage index is calculated as a test index value by comparing the number of covered attack test scenarios with the preset number of covered attack test scenarios. For instance, in the scenario of scenario coverage analysis of intelligent driving ECUs in vehicle cybersecurity verification, the performance evaluation method of the electronic control unit in this embodiment may be: during the vehicle cybersecurity verification stage, 12 covered attack scenarios are identified, while the preset total number of attack test scenarios to be covered is 15. By calculating the ratio of 12 / 15, the coverage rate is found to be 80%, and its integrity in attack surface protection can be evaluated based on this index value.

[0144] In the above embodiments, using the coverage of attack test scenarios as a test indicator value can more accurately reflect the functional implementation level of the electronic control unit when facing known threats.

[0145] In the aforementioned test method where the target test dimension is the coverage analysis dimension, the first number of completed test targets and the second number of preset test targets are determined from the ECU data. Based on the first and second numbers, the test index value of the target electronic control unit in the functional coverage analysis dimension is determined. By quantitatively evaluating the coverage of the electronic control unit to the preset test targets, specifically by statistically analyzing the ratio of the number of completed test targets to the total number of preset test targets, the corresponding test index value is generated. This can reflect the completeness of the electronic control unit's coverage of safety scenarios or functions in actual testing, thereby improving the comprehensiveness and accuracy of performance evaluation.

[0146] In an exemplary embodiment, implementation steps for weight allocation dimensions of the evaluation type are provided, specifically including:

[0147] If the assessment type is compliance type and the target test dimension is the weight allocation dimension, determine the type identifier of the target electronic control unit from the ECU data; determine the allocation weight corresponding to the type identifier based on the correspondence between the type identifier and the preset type identifier and allocation weight; determine the test index value of the target electronic control unit in the weight allocation dimension based on the allocation weight.

[0148] Understandably, for a vehicle, different types of ECUs have varying degrees of importance. For a less important ECU, even if it is compromised, it won't have a catastrophic impact on the vehicle. In other words, the cybersecurity risks of a vehicle are not evenly distributed. Different ECUs have different roles in the vehicle's functional and security architecture, and the severity of the consequences of their failure or compromise varies greatly. Therefore, when assessing overall vehicle security, different critical ECUs must be assigned different weights and weighted calculations must be performed. Type identifiers can be identification information used to identify the functional category or security level of the electronic control unit, and can be used as input for subsequent weight mapping. For example, it can be, but is not limited to, the name of the target electronic control unit.

[0149] For example, if the ECU is a gateway, the corresponding weight is 0.3; if the ECU is an ADAS, the corresponding weight is 0.25; if the ECU is an Infotainment, the corresponding weight is 0.2; if the ECU is a BCM, the corresponding weight is 0.15; and if the ECU is a TPMS, the corresponding weight is 0.1.

[0150] In the above embodiments, by determining the corresponding weights for different target ECUs based on their importance in the vehicle, the evaluation bias caused by uniform weights is avoided, thus ensuring the accuracy of electronic control unit performance evaluation.

[0151] In an exemplary embodiment, implementation steps for performance index dimensions of performance types are provided. Specifically, for each target test dimension, the corresponding target ECU data is determined from the ECU data, and the test index value of the target test dimension is determined based on the target ECU data, including:

[0152] If the evaluation type is performance-based and the target test dimension is performance index dimension, then the recall rate and false positive rate of the attack test of the target electronic control unit are determined from the ECU data; the recall rate and false positive rate are weighted according to the preset weighting formula to obtain the test index value of the target electronic control unit in the performance index dimension.

[0153] Recall, calculated as the number of detected real attacks divided by the total number of real attacks, measures the false negative rate. A higher recall indicates fewer attacks were missed. False positive rate (FPR), calculated as the number of false alarms divided by the total number of normal events, measures accuracy. A lower FPR indicates fewer false alarms.

[0154] The preset weighted formula can be expressed as 0.7×recall+0.3×(1-FPR). The weights of 0.7 and 0.3 in this formula are determined based on the characteristics of vehicle network security. Even if there is a false alarm, it must not be missed, because missing an attack may lead to catastrophic consequences.

[0155] In the above embodiments, by introducing a weighted calculation method of recall rate and false positive rate in the performance type evaluation, the test index values ​​under the performance index dimension can more comprehensively reflect the accuracy and stability of the intrusion detection system when identifying attacks.

[0156] The above embodiments provide implementation steps for different test dimensions of the evaluation type. The target test dimension of the evaluation type can include at least the above-mentioned at least two target test dimensions. Then, for any at least two target test dimensions, test results matching the evaluation type can be further generated based on multiple test indicator values.

[0157] In one exemplary embodiment, generating test results that match the evaluation type based on multiple test metric values ​​includes:

[0158] Based on at least one test indicator value, determine the weight corresponding to each test indicator value among the at least one test indicator values; based on each test indicator value and the weight corresponding to each test indicator value, determine the evaluation value corresponding to the evaluation type; based on the evaluation value, determine the test result that matches the evaluation type.

[0159] The multiple test metric values ​​can be from various test metric values ​​corresponding to the consistency analysis dimension, risk and vulnerability analysis dimension, coverage analysis dimension, weight allocation dimension, and performance metric dimension. The evaluation value can be an intermediate value used to generate the test result after combining multiple weighted test metric values. The evaluation type can include one or more of compliance type, performance type, etc. Optionally, when the evaluation type is compliance type, the weight value of each corresponding target test dimension is determined according to the test metric values ​​corresponding to the consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension; a dynamic weight matrix is ​​generated based on each weight value; and compliance data is generated based on the dynamic weight matrix.

[0160] The test metric values ​​and weight values ​​may have a pre-defined correspondence. The weight values ​​can be determined based on the historical performance, industry importance, or expert experience of each test dimension, assigning a corresponding weight value to each dimension. Compliance types may include, but are not limited to, one or more of the following: RR155 compliance, ISO26262 functional safety compliance, UNR157 cybersecurity compliance, and AUTOSAR standard compliance. The dynamic weight matrix can be a numerical matrix constructed for comprehensive evaluation based on the weight values ​​and / or test metric values ​​of each target test dimension. The dynamic weight matrix can perform weighted calculations on each test metric value and its corresponding weight value to form a matrix structure.

[0161] Based on the dynamic weight matrix, generating compliance data can involve comparing the comprehensive score of the dynamic weight matrix with a preset compliance threshold and outputting a conclusion or score on whether or not compliance is achieved.

[0162] For example, when the assessment type is compliance, the weight values ​​for each target test dimension are determined based on the corresponding test indicator values ​​in the consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension. A dynamic weight matrix is ​​generated based on these weight values. The dynamic weight matrix is ​​then matched with preset R155 regulatory provisions. Natural language generation technology is used to convert the quantitative results into structured report content. This structured content is then used to generate a corresponding R155 compliance report according to a preset template. In other words, the cybersecurity status of the ECU is quantitatively analyzed from four assessment dimensions: consistency, risk and vulnerability, functional coverage, and weight allocation. A dynamic weight matrix is ​​constructed to comprehensively evaluate the results of each dimension, ultimately automatically generating a compliance report that meets the requirements of UNECE R155 regulations. Furthermore, a corresponding R155 compliance report can also be generated based on the test indicator values ​​of the consistency analysis dimension according to a preset template.

[0163] In this approach, the weight values ​​for each target test dimension are determined based on the test indicator values ​​corresponding to the consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension. A dynamic weight matrix is ​​generated based on each weight value. Compliance data is then generated based on the dynamic weight matrix. This approach combines multi-dimensional test indicator values ​​to generate structured compliance data, improving the accuracy and interpretability of the assessment results and avoiding the one-sidedness caused by a single indicator assessment.

[0164] Optionally, in an exemplary embodiment, if the evaluation type is performance-based, the weights corresponding to the policy consistency analysis dimension, risk vulnerability analysis dimension, scenario coverage analysis dimension, and performance indicator dimension are obtained; each weight is weighted and aggregated with the test indicator values ​​of its corresponding target test dimension to obtain a quantified comprehensive risk assessment value; a preset weight coefficient for the target electronic control unit is determined; a security matrix is ​​generated based on the preset weight coefficient and the comprehensive risk assessment value; and an ECU attack scenario score is determined based on the security matrix.

[0165] The weights for the strategy consistency analysis dimension, risk vulnerability analysis dimension, scenario coverage analysis dimension, and performance metric dimension can be determined based on actual needs. For example, the weights for the strategy consistency analysis dimension, risk vulnerability analysis dimension, scenario coverage analysis dimension, and performance metric dimension could be 35%, 30%, 20%, and 15%, respectively.

[0166] The test index values ​​for different target test dimensions can be determined using the methods described above, which will not be elaborated upon here. Weighted aggregation can be achieved by multiplying the test index values ​​of each dimension by their corresponding weights and then summing them or performing other mathematical operations, thereby integrating the test results from multiple dimensions into a unified quantitative value. The comprehensive risk assessment value can be a numerical value reflecting the overall security risk level of the test object across multiple dimensions, and can be used as the basis for subsequently generating a security matrix.

[0167] The preset weighting coefficient can be a value pre-set based on factors such as the functional importance and safety priority of the target electronic control unit in the vehicle system. It can be used to reflect the relative position of the ECU in the vehicle's safety architecture. For example, if the ECU belongs to the power control type, the corresponding weighting coefficient is 1, and the safety threshold is the highest; if the ECU belongs to the autonomous driving type, the corresponding weighting coefficient is 0.9, and the safety threshold is high; if the ECU belongs to the body control type, the corresponding weighting coefficient is 0.8, and the safety threshold is medium-high; if the ECU belongs to the infotainment type, the corresponding weighting coefficient is 0.7, and the safety threshold is medium.

[0168] A security matrix can be a structured data model used to describe the risk distribution and impact of a target electronic control unit under different security dimensions, and can be used as the basis for subsequent attack scenario scoring.

[0169] For example, the system acquires ECU policy alignment index, CVE severity analysis results, standard coverage index, and detection rate / false alarm rate index for a target ECU under preset attack scenarios. Based on preset or dynamically adjusted weights, the system weights and aggregates these indicators to obtain a scenario score for the corresponding attack scenario. Then, a security matrix is ​​generated based on the security scores of each attack scenario, thereby obtaining the ECU's attack scenario score. It should be noted that each ECU has at least one attack scenario. Therefore, the score for a single attack scenario can be calculated based on the ECU policy alignment index, CVE severity analysis results, standard coverage index, and detection rate / false alarm rate index. By combining these with the weights of all vehicle ECUs, the overall security level of the vehicle is obtained.

[0170] Optionally, the security matrix can be scored based on four dimensions. The consistency analysis dimension is mainly based on the success rate of R155 test cases, with a maximum score of 100%. The higher the coverage, the higher the score; that is, the consistency test pass rate equals the score for this dimension. The risk and vulnerability analysis dimension is mainly based on penetration test cases, matched test cases, and functional vulnerabilities. Points are deducted according to the severity of different vulnerabilities. For example, the vulnerability deduction principle is 3 points for low risk, 5 points for medium risk, 8 points for high risk, and 10 points for severe vulnerability. The initial maximum score for this dimension is 100. It should be noted that this scoring is for a single ECU. If the entire vehicle is considered, a weighting coefficient for each ECU needs to be added. Scenario coverage analysis dimension: Based on the ECUs provided by the supplier, determine whether the corresponding functional test cases are met. If fully met, it is 100%. This is used as the weight according to the proportion. Then, the known function success rate (maximum score 100%) is added to the weight to obtain the final score. Building upon this foundation, to determine the overall vehicle score, a weighting dimension can be further considered. This weighting dimension serves as a comprehensive scoring standard, with different types of ECUs having varying importance and corresponding weights. For example, the Gateway has a weight of 0.3, the DAS (Advanced Driver Assistance System) 0.25, the Infotainment system 0.2, the BCM (Body Control Module) 0.15, and the TPMS (Tire Pressure Monitoring System) 0.1, with a total weight of 1. A comprehensive score is then calculated based on the actual number of ECUs included in the vehicle. This comprehensive score is determined by the weighting ratios and other dimension scores, resulting in the final overall score. For example, the overall vehicle safety score = Gateway score × 0.30 + ADAS score × 0.25 + Infotainment score × 0.20 + BCM score × 0.15 + TPMS score × 0.10. Furthermore, based on this embodiment, performance indicator scores can also be considered to determine the attack scenario scores for individual ECUs and the entire vehicle.

[0171] In this approach, by obtaining the weights corresponding to multiple test dimensions, the weights are weighted and aggregated with the test index values ​​to generate corresponding risk assessment values. A safety matrix is ​​generated based on each risk assessment value, and the attack scenario score of a single ECU and the whole vehicle can be determined based on the safety matrix, thus improving the accuracy of the attack scenario score.

[0172] In the above embodiments, by determining the weight corresponding to each test indicator value based on multiple test indicator values, determining the evaluation value corresponding to the evaluation type based on each test indicator value and its corresponding weight, and determining the test result matching the evaluation type based on the evaluation value, it is possible to differentiate the importance of different test indicators in the overall performance and improve the accuracy of the evaluation.

[0173] It should be noted that the above embodiments provide implementation steps for different test dimensions of the evaluation type. The target test dimension of the evaluation type can include at least two of the above-mentioned target test dimensions. The following provides a performance evaluation method for electronic control units with four test dimensions of the evaluation type, such as... Figure 6 As shown, it includes:

[0174] When the assessment type is compliance type, obtain the ECU data of the target electronic control unit corresponding to the assessment type, and obtain the consistency analysis dimension, risk vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension that match the assessment type from the preset test dimensions. For each target test dimension in the consistency analysis dimension, risk vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension, determine the corresponding target ECU data from the ECU data, and determine the test index value of the target test dimension based on the target ECU data.

[0175] Based on the test metric values ​​corresponding to the consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension, the weight values ​​for each corresponding target test dimension are determined. A dynamic weight matrix is ​​generated based on these weight values, and compliance data is generated based on the compliance threshold and the dynamic weight matrix. Furthermore, an R155 compliance report can be generated based on the compliance data.

[0176] It should be noted that the specific implementation steps of this example can be achieved in the manner described above, and will not be repeated here.

[0177] In the above embodiments, for each compliance type, the weight value of the corresponding target test dimension is determined based on the test indicator values ​​of the consistency analysis dimension, risk and vulnerability analysis dimension, functional coverage analysis dimension, and weight allocation dimension. A dynamic weight matrix is ​​generated based on each weight value. Compliance data is then generated based on the dynamic weight matrix. This approach, by introducing multi-dimensional test indicators and combining the collaborative analysis of different test dimensions, generates a comprehensive evaluation result, avoiding the limitations of a single detection rate indicator and thus improving the accuracy of electronic control unit performance evaluation. Furthermore, different target test dimensions can be selected according to different needs, supporting the requirements of various application scenarios.

[0178] In one exemplary embodiment, a performance evaluation method for an electronic control unit with four test dimensions of performance type is provided, such as... Figure 7 As shown, it includes:

[0179] When the assessment type is performance-based, input parameters are obtained. These input parameters include the target ECU's attribute information. Based on the ECU's function type in the attribute information, the corresponding R155 threat level is determined, and the corresponding attack complexity and ECU data are determined based on environmental parameters. The R155 threat level and attack complexity can be determined using existing methods, which will not be elaborated upon here. Attribute information may include ECU function type, current security policy version, compliance requirements, and environmental parameters.

[0180] Dynamic weights are calculated based on the R155 threat level and attack complexity, and the weights of each dimension in the four-dimensional evaluation model are dynamically allocated, that is, the weights of the policy consistency analysis dimension, risk vulnerability analysis dimension, scenario coverage analysis dimension, and performance indicator dimension are determined.

[0181] For the strategy consistency analysis dimension, risk vulnerability analysis dimension, scenario coverage analysis dimension, and performance indicator dimension, the test indicator values ​​on their respective target test dimensions are determined, namely, the ECU strategy alignment, CVE severity analysis, standard coverage, and detection rate / false alarm rate are calculated respectively.

[0182] Obtain the weights corresponding to the strategy consistency analysis dimension, risk vulnerability analysis dimension, functional coverage analysis dimension, and performance indicator dimension; weight and aggregate each weight with the test indicator value of its corresponding target test dimension to obtain a quantified comprehensive risk assessment value; determine the preset weight coefficients of the target electronic control unit, and generate a security matrix based on the preset weight coefficients and the comprehensive risk assessment value; determine the ECU attack scenario score based on the security matrix.

[0183] It should be noted that the specific implementation of this embodiment can be achieved in the manner defined above, and will not be elaborated here.

[0184] In this embodiment, for ECU attack scenario scoring, the weights corresponding to the strategy consistency analysis dimension, risk vulnerability analysis dimension, functional coverage analysis dimension, and performance indicator dimension are obtained. Each weight is then weighted and aggregated with the test indicator values ​​of its corresponding target test dimension to obtain a quantified comprehensive risk assessment value. A preset weight coefficient for the target electronic control unit is determined, and a security matrix is ​​generated based on the preset weight coefficient and the comprehensive risk assessment value. The ECU attack scenario score is then determined based on the security matrix. This approach, by introducing multi-dimensional test indicators and combining collaborative analysis of different test dimensions, generates a comprehensive evaluation result, avoiding the limitations of a single detection rate indicator and improving the accuracy and reliability of the ECU attack scenario score.

[0185] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0186] Based on the same inventive concept, this application also provides a performance evaluation device for an electronic control unit (ECU) to implement the performance evaluation method for the ECU described above. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations in one or more ECU performance evaluation device embodiments provided below can be found in the limitations of the ECU performance evaluation method described above, and will not be repeated here.

[0187] In one exemplary embodiment, such as Figure 8 As shown, a performance evaluation device for an electronic control unit is provided, comprising: a data acquisition module 802, a test dimension determination module 804, a test module 806, and an evaluation module 808, wherein:

[0188] The data acquisition module 802 is used to acquire ECU data of the target electronic control unit under the desired evaluation type.

[0189] The test dimension determination module 804 is used to determine at least one target test dimension that matches the evaluation type based on preset test dimensions.

[0190] Test module 806 is used to find the target ECU data corresponding to the evaluation type from the ECU data for each target test dimension, and determine the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data.

[0191] Evaluation module 808 is used to generate test results that match the evaluation type based on at least one test indicator value.

[0192] The aforementioned electronic control unit performance evaluation device acquires ECU data of the target electronic control unit corresponding to the evaluation type, obtains multiple target test dimensions matching the evaluation type from preset test dimensions, determines the corresponding target ECU data from the ECU data for each target test dimension, determines the test index value of the target test dimension based on the target ECU data, and generates test results matching the evaluation type based on multiple test index values. This method, by introducing multi-dimensional test indicators and combining the collaborative analysis of different test dimensions, generates a comprehensive evaluation result, avoiding the limitations of a single detection rate indicator, thereby improving the accuracy of electronic control unit performance evaluation.

[0193] In an exemplary embodiment, the test dimension determination module 804 is further configured to determine, when the assessment type is a compliance type, a plurality of target test dimensions that match the assessment type, including at least one of the following: consistency analysis dimension, risk and vulnerability analysis dimension, coverage analysis dimension, and weight allocation dimension;

[0194] When the evaluation type is performance-based, the multiple target test dimensions that match the evaluation type should include at least one of the following: consistency analysis dimension, risk and vulnerability analysis dimension, coverage analysis dimension, and performance metric dimension.

[0195] In an exemplary embodiment, the test module 806 is further configured to search for the local data of the target electronic control unit and the associated data of the associated electronic control units of the target electronic control unit from the ECU data when the target test dimension is the consistency analysis dimension;

[0196] The difference value is determined based on local data and related data; the difference value is used to characterize the degree of difference between local data and related data.

[0197] The test index values ​​of the target electronic control unit in the consistency analysis dimension are determined based on the difference value.

[0198] In an exemplary embodiment, the testing module 806 is further configured to determine the vulnerability exposure factor and the general vulnerability scoring system score corresponding to the identified vulnerability from the ECU data when the evaluation type is performance type and the target testing dimension is risk vulnerability analysis dimension.

[0199] For each identified vulnerability, a risk score is determined based on the vulnerability exposure factor and the score from the general vulnerability scoring system to characterize the risk level of the identified vulnerability.

[0200] The test index values ​​for the target electronic control unit in the risk vulnerability analysis dimension are determined based on the risk scores of all identified vulnerabilities.

[0201] In an exemplary embodiment, the test module 806 is further configured to determine, when the target test dimension is a coverage analysis dimension, a first number of completed test targets and a second number of preset test targets from the ECU data; the test targets are covered attack test scenarios or security functions.

[0202] The test index values ​​of the target electronic control unit in the functional coverage analysis dimension are determined based on the first and second quantities.

[0203] In an exemplary embodiment, the testing module 806 is further configured to search for vulnerability types from ECU data if the assessment type is a compliance type and the target testing dimension is a risk vulnerability analysis dimension.

[0204] For each vulnerability type, the risk level and risk score corresponding to that vulnerability type are determined based on the vulnerability type, the preset correspondence between vulnerability type, risk level and risk score.

[0205] Based on the risk level and risk score corresponding to all vulnerability types, determine the target risk score corresponding to the target vulnerability type with the highest risk level.

[0206] The test index values ​​of the target electronic control unit in the risk vulnerability analysis dimension are determined based on the target risk score.

[0207] In an exemplary embodiment, the test module 806 is further configured to determine the type identifier of the target electronic control unit from the ECU data if the evaluation type is a compliance type and the target test dimension is a weight allocation dimension;

[0208] The allocation weight corresponding to the type identifier is determined based on the correspondence between the type identifier and the preset type identifier and allocation weight;

[0209] The test index values ​​of the target electronic control unit in the weight allocation dimension are determined based on the assigned weights.

[0210] In an exemplary embodiment, the evaluation module 808 is further configured to determine the weight corresponding to each of the at least one test index values ​​based on at least one test index value.

[0211] The evaluation value corresponding to the evaluation type is determined based on the values ​​of each test indicator and the weights corresponding to each test indicator.

[0212] The test results that match the evaluation type are determined based on the evaluation values.

[0213] Each module in the performance evaluation device for the aforementioned electronic control unit can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0214] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores ECU data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a performance evaluation method for an electronic control unit.

[0215] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0216] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0218] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0219] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0220] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0221] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0222] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A performance evaluation method for an electronic control unit, characterized in that, The performance evaluation method includes: According to the desired evaluation type, obtain the ECU data of the target electronic control unit under the evaluation type; Based on the preset test dimensions, determine at least one target test dimension that matches the evaluation type; For each target test dimension, the target ECU data corresponding to the evaluation type is found from the ECU data, and the test index value corresponding to the evaluation type for the target test dimension is determined based on the target ECU data. Generate test results that match the evaluation type based on at least one of the test metric values.

2. The method according to claim 1, characterized in that, The step of determining at least one target test dimension that matches the evaluation type based on preset test dimensions includes: When the assessment type is a compliance type, the target testing dimension that matches the assessment type shall include at least one of the following: consistency analysis dimension, risk and vulnerability analysis dimension, coverage analysis dimension, and weight allocation dimension. When the evaluation type is performance type, the target test dimension that matches the evaluation type shall include at least one of the consistency analysis dimension, the risk and vulnerability analysis dimension, the coverage analysis dimension, and the performance indicator dimension.

3. The method according to claim 1 or 2, characterized in that, The step of generating test results matching the evaluation type based on at least one of the test metric values ​​includes: Based on at least one of the test indicator values, determine the weight corresponding to each of the at least one test indicator values; The evaluation value corresponding to the evaluation type is determined based on each test indicator value and the weight corresponding to each test indicator value. The test result matching the evaluation type is determined based on the evaluation value.

4. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: When the target test dimension is the consistency analysis dimension, the local data of the target electronic control unit and the associated data of the associated electronic control units are retrieved from the ECU data. A difference value is determined based on the local data and the associated data; the difference value is used to characterize the degree of difference between the local data and the associated data. The test index value of the target electronic control unit in the consistency analysis dimension is determined based on the difference value.

5. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: When the evaluation type is the performance type and the target test dimension is the risk vulnerability analysis dimension, the vulnerability exposure factor and the general vulnerability scoring system score corresponding to the identified vulnerability are determined from the ECU data. For each identified vulnerability, a risk score is determined based on the vulnerability exposure factor and the score from the general vulnerability scoring system to characterize the risk level of the identified vulnerability. The test index value of the target electronic control unit in the risk vulnerability analysis dimension is determined based on the risk scores of all identified vulnerabilities.

6. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: When the target test dimension is the coverage analysis dimension, a first number of completed test targets and a second number of preset test targets are determined from the ECU data; the test targets are the covered attack test scenarios or security functions. The test index value of the target electronic control unit in the functional coverage analysis dimension is determined based on the first quantity and the second quantity.

7. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: If the assessment type is the compliance type and the target testing dimension is the risk vulnerability analysis dimension, then search for the vulnerability type in the ECU data; For each vulnerability type, the risk level and risk score corresponding to that vulnerability type are determined based on the vulnerability type, the preset correspondence between vulnerability type, risk level and risk score. Based on the risk level and risk score corresponding to all vulnerability types, determine the target risk score corresponding to the target vulnerability type with the highest risk level. The test index value of the target electronic control unit in the risk vulnerability analysis dimension is determined based on the target risk score.

8. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: If the assessment type is the compliance type and the target test dimension is the weight allocation dimension, determine the type identifier of the target electronic control unit from the ECU data; The allocation weight corresponding to the type identifier is determined based on the correspondence between the type identifier and the preset type identifier and allocation weight; The test index value of the target electronic control unit in the weight allocation dimension is determined based on the allocated weights.

9. The method according to claim 2, characterized in that, For each target test dimension, the process involves retrieving the target ECU data corresponding to the evaluation type from the ECU data, and determining the test index value corresponding to the evaluation type for the target test dimension based on the target ECU data, including: If the evaluation type is performance type and the target test dimension is the performance indicator dimension, then the recall rate and false alarm rate of the attack test of the target electronic control unit are determined from the ECU data. The recall rate and false alarm rate are weighted according to a preset weighting formula to obtain the test index value of the target electronic control unit in terms of performance indicators.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images