Method for adapting test cases for a security inspection
Patent Information
- Application Number
- EP2023837126
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-12-23
- Filing Date
- 2023-12-22
- Publication Date
- 2025-10-29
AI Technical Summary
The complexity of functional systems in the automotive and aeronautics sectors creates a large attack surface for cyber attacks, making it difficult to create standardized test cases for security checks, requiring manual adaptation for each system, which is time-consuming and resource-intensive.
A computer-implemented method to adapt existing test cases for security checks by dividing them into modules, abstracting system-specific information, creating abstract test scenarios, and deriving new test cases for specific systems, allowing for automated reuse and adaptation of test cases across different functional systems.
This method enables the efficient and resource-saving adaptation of existing test cases for new systems, reducing the need for manual adaptation and saving time, while providing feedback for improving test scenario effectiveness.
Smart Images
Figure 1.1
Abstract
Description
[0001] Procedure for adapting test cases for a security review
[0002] Technical area
[0003] The present invention relates to a computer-implemented method for adapting test cases for a safety check of a functional system to be tested in a mobility application, in particular from the automotive and / or aeronautics sector, using a test system. For this purpose, test cases are provided to the test system that have been used for safety unit checks of functional systems used in or for mobility applications. Furthermore, the present invention relates to an associated test system and an associated computer program product.
[0004] State of the art
[0005] Many modern technical devices, particularly vehicles, have a certain degree of complexity and comprise a multitude of embedded and interconnected components, each equipped with its own processors or microcontrollers. However, the present disclosure is not limited to vehicles in the true sense of the word, but can also be used in conjunction with other technical units and systems that comprise a multitude of interconnected components and are used primarily in the automotive and / or aeronautics sectors in mobility applications—i.e., for example, in vehicles or in their surroundings. In the context of the present disclosure, these are generally summarized under the term “functional system.” This means that such a functional system can, for example, be a vehicle itself (e.g., motor vehicle, truck, aircraft, etc.).) or a complex system that is used in a vehicle or in conjunction with a vehicle to implement mobility applications.
[0006] Examples of such functional systems include, in addition to the vehicles themselves, combinations of different external units / systems with vehicles or internal vehicle units and systems, combinations of internal vehicle systems and units, and combinations of multiple vehicles or units / systems of multiple vehicles. For example, certain tasks can be carried out by an autonomous computer unit (e.g. tablet, smartphone, etc.), which communicates via an interface with a unit or system in an (autonomous) vehicle (e.g. infotainment system, GPS, navigation system, etc.). In mobility applications such as autonomous driving (AD) or so-called vehicle-to-X communication, vehicles or internal vehicle systems can exchange data or information with infrastructure units (e.g. traffic lights, barrier systems, etc.), for example via radio. Furthermore, in mobility applications such asFor example, in driver assistance systems or so-called Advanced Driving Assistance Systems (ADAS for short), units and systems (e.g. sensors, etc.) within a vehicle communicate with other units and systems (e.g. braking system, engine, etc.) via interfaces. On the other hand, several vehicles can also communicate with each other, for example via radio, in order to carry out tasks together - for example by being linked together like a virtual drawbar, or to implement mobility applications such as vehicle-to-vehicle communication for the exchange of information and data between vehicles. Furthermore, a vehicle, such as a special vehicle in the construction sector, in agriculture, an emergency vehicle, etc., can be connected to one or more on-board devices.Such combined units or systems are considered, within the meaning of the present disclosure, to be a functional system that is used in a mobility application (e.g., vehicle, autonomous driving, ADAS, vehicle-to-infrastructure or vehicle-to-vehicle applications, etc.) or is used for the implementation thereof.
[0007] Typically, the components of such a functional system have their own memory units and communication interfaces, thus each forming its own computer system. Such components are also referred to as "embedded systems (ES)" and perform their tasks – largely invisibly – within the functional system. In the case of a complex overall system (e.g., vehicle, aircraft, etc.), a large number of otherwise autonomous embedded systems are usually networked. Components of such functional systems can also be so-called "cyber-physical systems (CPS). A cyber-physical system is typically understood to be a network of software units with mechanical and electronic parts, which are connected via a data infrastructure (e.g., Internet, bus system, etc.).), whereby a cyber-physical system can be formed, for example, by networking embedded systems via a wired and / or wireless communication network and can be used, for example, in mobility applications (e.g. networked security systems, networked driver assistance systems, etc.).
[0008] The complexity of such functional systems, especially in the automotive and / or aeronautics sectors, quickly becomes unmanageable. For example, a vehicle typically has dozens of control units, each executing software with tens of millions of lines of code. Examples of communication interfaces include the various wireless connection protocols, such as mobile communications protocols (e.g., 5G, LTE, etc.), Bluetooth, Wireless LAN, RFID, Vehicle-to-X interfaces, etc., which are sometimes used simultaneously. Due to the complexity and diverse communication interfaces, such functional systems create a large attack surface for cyberattacks, such as denial-of-service (DoS) attacks / flooding.Furthermore, there is a risk that attacks will occur not only via the actual communication interfaces, but also via sensors (e.g., LIDAR, radar systems, etc.), for example, whereby such attacks can be carried out using so-called fake signals. In principle, any event that occurs from outside or inside the functional system with the aim of influencing the functional system in an impermissible manner and / or of impermissibly forwarding data from the functional system to third parties is considered a cyberattack.
[0009] Due to the physical capabilities of vehicles (mass and speed, thus high kinetic energy, direct interaction with potentially large numbers of people), their large number (vehicle fleets), and the development toward autonomous vehicles, such attacks can pose an enormous threat potential. This applies not only to the large number of vehicles, such as cars or trucks, but also affects rail vehicles, watercraft, and, above all, aircraft. Therefore, it is important to protect functional systems used in mobility applications—especially in the automotive and aeronautics sectors—against cyberattacks and thus misuse, so that such attacks on the respective system have as little effect as possible or remain as ineffective as possible.
[0010] To protect functional systems against cyberattacks and make them resilient, vulnerabilities must be identified and, if possible, prevented as early as possible in the development process. In the concept phase, this can be achieved through appropriate architectural measures, such as those described in the publication "Secure Vehicular Communication Systems: Design and Architecture" by Papadimitratos, P., Buttyan, L., Holczer, T., Schoch, E., Freudiger, J., Raya, M., Ma, Z., Kargl, F., Kung, A., & Hubaux, J.-P., 2008, IEEE Communications Magazine, 46(11), 100-109. During concept implementation, this can be achieved, for example, through appropriate development processes such as best practices and source code reviews.
[0011] After complete assembly and integration of the components into the functional system, a security review of the functional system is always required to identify as many relevant vulnerabilities as possible – especially those that arise through the combination of components, such as embedded and / or cyber-physical systems. The security review is intended to enable manufacturers (original equipment manufacturers - OEMs), government agencies (e.g., regulatory authorities), and other stakeholders (e.g., consumer associations, commercial fleet operators, etc.) to assess the specific risk of a functional system of a mobility application, such as a vehicle, with regard to cyber attacks. This can be used in the further development or improvement of the vehicles or the functional system, for acceptance or certification, and similar tasks.The security audit is intended to uncover existing, but initially mostly unknown, vulnerabilities and potential points of attack for cyberattacks, particularly in functional systems (e.g., vehicles, etc.). To uncover vulnerabilities and potential points of attack (i.e., initially unrecognized paths through which a cyberattack on the system under test could be carried out), a security audit of a functional system defines various attacks (e.g., in the form of test vectors) on the functional system under test. These attacks are then applied to the functional system under test as system-specific test cases within the framework of the security audit.
[0012] For example, safety checks are required during the development of a new vehicle at the OEM, during market launch / type approval / homologation, and subsequently continuously throughout the entire life cycle of the vehicle or the respective functional system. The need for repeated safety checks at the system test level and thus an ongoing cybersecurity review arises due to changeable configurations, ongoing updates of (parts of) software components, changed environmental conditions (e.g., vehicle-to-infrastructure - V2I, vehicle-to-vehicle - V2V: changes in interaction partners), newly discovered test vectors (including from in-house safety research), etc. In addition to the manufacturer, the safety check can also be carried out by (fleet) operators, regulatory authorities, and / or specialized third-party companies.
[0013] However, due to very restrictive protection and confidentiality policies of manufacturers, especially in the automotive sector, but also in the aeronautics sector, and / or due to the individualized and / or manufacturer-specific design of functional systems for mobility applications, it is generally the case that no consistently standardized hardware and software systems are used for functional systems. It is therefore difficult to create generally applicable test cases for security checks that are suitable for making statements about the security or cybersecurity of functional systems with the same functionality, e.g. from different manufacturers. Even updates to components of a functional system or an update of a functional system can lead to new test cases having to be created for the security check or at least existing test cases having to be adapted. This means that the test cases relating to security orThe functional systems to be tested for cyber security are often heterogeneous and largely manufacturer-specific - even with the same functionality - so that the test cases for security checks must be system-specific and individually created, or at least adapted, for each functional system to be tested. Vulnerabilities and potential points of attack may be similar, but due to the individual design of the functional system to be tested, existing test cases cannot be directly transferred from a functional system that has already been tested to a functional system that is yet to be tested. This means that the test cases for the security check must be individually and often manually adapted to the functional system to be tested. This makes security checks of functional systems, such asVehicles, parts or subsystems of vehicles used in mobility applications involve a great deal of time and resources.
[0014] For example, US 2005 / 0160322 A1 discloses a method and system by which a specific automation test script for detecting errors in an application is converted into an abstract test case representation. The abstract test case representation is then stored in a database for reuse. One or more application states, external interaction sequences, and input data are stored for the abstract test case representation, and the abstract test representation can be supplemented with information from an application metadata repository so that it can be used to test the application in other test environments or on other test platforms. Although this allows test cases to be abstracted for testing an application, the abstract test case representation can only be used to test the same application in a different test environment or on a different test platform.
[0015] Description of the invention
[0016] The invention is therefore based on the object of specifying a method by which test cases which have already been used in security checks for testing functional systems and / or which can be used for such security checks can be adapted and reused for security checks of further functional systems which are still to be tested in a simple, resource-saving and automated manner.
[0017] These and other objects are achieved by a method according to the independent claim. Advantageous embodiments of the present invention are described in the dependent claims. According to the invention, the object is achieved by a computer-implemented method for adapting test cases for a security check of a functional system used in a mobility application, using a test system. The test cases that have already been applied to security checks on functional systems used in mobility applications and / or are applicable for such security checks are made available to the test system. The following steps are carried out:
[0018] Dividing the test cases into test modules;
[0019] - Abstracting the test modules, whereby system-specific information, data and parameters of the functional systems to which the respective test cases were applied, contained in the test modules, are replaced by abstract placeholder variables in a respective test module;
[0020] Creating abstract test scenarios for the security testing of the functional system to be tested, whereby the abstracted test modules are combined for one test scenario;
[0021] - Deriving test cases for the functional system to be tested from the abstract test scenarios, whereby the respective abstract placeholder variables in the abstracted test modules of a respective test scenario are assigned corresponding system-specific information, data and parameters of the functional system to be tested; and
[0022] - Applying the test cases derived from the test scenarios to the functional system to be tested.
[0023] The main aspect of the proposed solution is that, in a simple and resource-efficient manner, already known test cases or those already used and / or usable for safety checks of functional systems in mobility applications can be generalized and made usable for safety checks of newly tested functional systems. The adaptation to the respective new functional system to be tested is carried out automatically. The method according to the invention automatically "recycles" previously used test cases for safety checks of newly tested functional systems.This means that new test scenarios and specific test cases do not have to be developed for each new functional system to be tested. Instead, test cases that have already been used in security tests on other functional systems and / or are applicable for them can be used and then automatically adapted for the security test of the new functional system to be tested. This saves resources and time during the security test. Furthermore, it is advantageous if, after the test cases derived from the test scenarios have been applied to the functional system to be tested, a feedback message is created for each test case. The feedback is then transmitted to the test system and linked in the test system to the respective test scenario from which the respective test case was derived.This makes it very easy to identify, for example, particularly well-functioning test scenarios or poorly functioning test scenarios. The feedback also allows for the recording of each test scenario, for which functional system it has already been applied, and how helpful it was, for example, in uncovering security-relevant vulnerabilities. This allows for subsequent security audits of the same or a similar functional system to quickly identify functioning test scenarios and avoid poorly functioning test scenarios.
[0024] It is also advantageous if, during abstraction, the test modules are searched for required parameters which are to be made available by the functional system to be tested or at least one of the preceding test modules in order to carry out the respective test module. These required parameters can be stored in a parameter list in the test system, whereby the respective parameter list is linked to the test module abstracted from the respective test module. Furthermore, relations between individual parameters can also be saved in the parameter lists in order to be able to use them when creating test scenarios. Using the parameter lists, test scenarios which are not useful, do not function and / or do not achieve the desired result can also be quickly identified and eliminated when creating test scenarios, for example if, due to a sequence and / or combination of test modules, etc., required parameters, for example,are not available. Furthermore, the parameter lists can be used to include additional steps in the test scenario to determine these parameters—particularly when applying the test case derived from the respective test scenario. The creation and execution of these steps, such as determining the parameter or parameter value from the functional system to be tested, etc., can be automated, for example.
[0025] It is expedient to search a created, abstract test scenario for placeholder variables of required parameters, which are to be provided to the respective test module by the functional system to be tested and / or by at least one of the preceding test modules for the execution of the test case derived from the test scenario.
[0026] Ideally, the parameter lists associated with the respective abstracted test modules of the respective test scenario are used to locate the placeholder variables for the required parameters in the respective test scenario. This allows the placeholder variables and, if applicable, the relationships between them to be quickly and easily identified and used.
[0027] Furthermore, it is advantageous if the abstracted test modules and / or abstract test scenarios and / or parts of abstract test scenarios are stored in the test system, in particular in a database. This allows them to be reused for subsequent security checks of functional systems to be tested. On the one hand, test cases can then be adapted more quickly to newly tested functional systems, and on the other hand, a test module and scenario database is created, thus further accelerating the inventive method. It can also be advantageous to check whether the respective abstracted test module is already stored in the test system, in particular in the database, before storing the respective abstracted test module. This very easily prevents abstracted test modules from being saved multiple times.
[0028] Ideally, information is stored in the test system for each abstract test scenario, indicating which functional system the respective test scenario can be applied to. This allows suitable test scenarios to be found quickly and with minimal effort for security checks of functional systems under test.
[0029] A suitable embodiment of the method provides that the system-specific information, data, and parameters of the functional system to be tested for assigning the placeholder variables in the abstracted test modules of the respective test scenarios are provided by the test system, in particular by the database in which empirical values from previous security checks of an identical or similar functional system are stored, and / or by a system manufacturer, and / or are determined by means of machine learning from empirical values from previous security checks. However, values determined from an analysis of the respective functional system to be tested can also be used as system-specific information, data, and parameters of the functional system to be tested for assigning the placeholder variables in the abstracted test modules of the respective test scenarios.
[0030] The above-mentioned object is further achieved by a test system and a computer product. The test system has at least one computer unit configured to carry out the method according to the invention. The test system can expediently also comprise a storage unit, such as a database, or be connected to a storage unit, such as a database. This storage unit is ideally configured to store the abstracted test modules, the abstract test scenarios and / or parts of abstract test scenarios, as well as empirical values from previous security checks of the same or similar functional system.The test system can also be connected to or have a machine learning system with which the system-specific information, data and parameters of the functional system to be tested can be determined from empirical values, etc. to populate the placeholder variables in the abstracted test modules of the respective test scenarios.
[0031] The computer program product can advantageously be loaded into the test system - for example into the at least one computer unit - and comprises instructions which, when the computer program product is executed in the test system, cause the computer program product to carry out the method according to the invention.
[0032] Short description of the characters
[0033] The present invention will be explained in more detail below with reference to Figures 1 to 2c, which show exemplary, schematic and non-limiting advantageous embodiments of the invention.
[0034] Fig.1 shows a flow of the computer-implemented method for adapting test cases of a safety test of a functional system to be tested
[0035] Fig.2a an abstraction step of the method according to the invention based on a concrete embodiment
[0036] Fig.2b shows a synthesis step of the method according to the invention based on the specific embodiment; and
[0037] Fig.2c shows a concretization step of the method according to the invention based on the concrete embodiment.
[0038] Implementation of the invention
[0039] Figure 1 shows an example sequence of a computer-implemented method with which test cases TC1, which have already been applied in safety checks of functional systems by which, for example, mobility applications (e.g. motor vehicles, etc.) are realized and / or which are used in mobility applications (e.g. autonomous driving, vehicle-to-vehicle communication, vehicle-to-X communication, etc.), can be automatically adapted for a safety test of a functional system to be tested (e.g. vehicle, vehicle system, etc.). The functional system to be tested can be a newly developed functional system, a further development and / or improvement of an existing functional system or even a functional system in which, for example, individual components (e.g. hardware units, software units, etc.) have been replaced or updated.
[0040] The method can be executed on an associated test system, wherein the test system has at least one computer unit. The test system represents an active system that performs the security check of a system to be tested according to a computer program product, which can be loaded, for example, into an internal memory of the test system and executed, for example, by means of the at least one computer unit or a processor. For this purpose, the computer program product comprises instructions which, when executed by the test system, cause the computer program product to perform the steps of the method according to the invention.
[0041] The test system executes corresponding test cases TC1, TC21, ..., TC23, for example, to uncover vulnerabilities in the functional system under test. For this purpose, the test system can, for example, provide interfaces and an environment (e.g., a test bench or testbed for mobility applications, etc.) and, together with the functional systems under test, form a test environment in which the inventive method is then carried out. The test environment or, respectively, the test system contains test cases TC1 that have already been applied to security checks of other functional systems. This means that the test system knows test cases TC1 that have already been applied to security checks of functional systems used in mobility applications and / or that can be used for such security checks. A test case TC1, TC21, ..., TC23 is a test scenario TS1, TS2, TS3 adapted to the respective functional system to be tested and can, for example, be executed on the test system and the functional system to be tested as part of the security review. In the context of this disclosure, a test scenario TS1, TS2, TS3 is seen as an abstract description of a test case TC1, TC21, ..., TC23, which defines what is to be tested and with which means, and in which individual test scenario phases can be described. Test scenarios TS1, TS2, TS3 for functional systems can, for example, be obtained from security analyses and / or from security requirements.
[0042] In order to adapt a specific test case TC1, which has already been applied to a functional system using the test system, for another functional system that is currently to be tested, the test system carries out the steps described below and shown as examples in Figure 1. For this purpose, in a modularization step 101, test cases TC1 already used to test functional systems are analyzed by the test system and divided or broken down into individual test modules T11, T12, T13, T14. A test module T11, T12, T13, T14 can, for example, be a test script which, when executed, represents a targeted attack on the functional system. If necessary, the test module T11, T12, T13, T14 can also consist of just one or more commands.
[0043] In a subsequent abstraction step 102, the test modules T11, T12, T13, and T14 determined from the test cases TC1 are then abstracted. In the process, the system-specific information, data, and parameters of the functional system to which the respective test case TC1 was applied contained in the test modules T11, T12, T13, and T14 are replaced by abstract placeholder variables. Thus, in the abstraction step 102, abstracted test modules a1, a2, a3, and a4 are created from the concrete test modules T11, T12, T13, and T14, which are system-specifically adapted for a respective test case TC1. An abstracted test module a1, a2, a3, a4 thus represents a system-agnostic test module, which describes a single phase of a test scenario TS1, TS2, TS3 and is agnostic with respect to a specific functional system.
[0044] The abstracted test modules a1, a2, a3, a4 can be stored in the test system, e.g., in an associated database. The database can be located, for example, in an internal storage unit of the test system or in a storage unit that is connected to the test system. Before storing the respective abstracted test module a1, a2, a3, a4, it can be checked whether this test module a1, a2, a3, a4 is already stored in the test system or in the associated database. In order to be able to store the abstracted test modules a1, a2, a3, a4 in an adequate form in the test system or in the database, an appropriate description language - such as a so-called Domain Specific Language, or DSL for short - is used for the abstracted test modules a1, a2, a3, a4. A Domain Specific Language is a formal language that is used for interactions between, for example, people and digitally working units (e.g.,Computer) is designed and implemented for a specific problem area - the so-called domain.
[0045] Furthermore, in the abstraction step 102, the test modules T11, T12, T13, T14 can be searched for required parameters before abstraction. Required parameters are, for example, parameters that are absolutely necessary for the meaningful execution of the respective test module T11, T12, T13, T14. These parameters can, for example, be provided by the functional system to be tested or by one of the preceding test modules T11, T12, T13, T14 in the respective test case TC1. However, they can also originate from an external source, such as a database, machine-readable specification, etc., or be entered as manual input. Such parameters would be, for example, network addresses of a test target, an identification and / or a specific mapping address, etc., which must be present when sending a specific command via or to a component of the functional system under test for the execution of the test module T11, T12, T13, T14, or configuration data. These parameters define, for example, properties (in one step) of the respective test module T11, T12, T13, T14 - such as the network address or port number to which a specific message should be sent, or how long this message should be.
[0046] Required parameters can occur in every test module T11, T12, T13, T14 and can be specific to that particular test module T11, T12, T13, T14. Furthermore, there may also be required parameters that are necessarily the same for all test modules T11, T12, T13, T14, e.g., of a test case TC1, or even other test cases TC21, TC22, TC23 for the respective functional system, such as the port number. Such parameters can, for example, be defined once for all test modules T11, T12, T13, T14 of a test case TC1 as "global" parameters.
[0047] The determined, required parameters of the respective test module T11, T12, T13, T14 can then be summarized in a parameter list, for example. In addition to the determined, required parameters of the respective test module T11, T12, T13, T14, relationships between individual, required parameters can also be saved in the parameter list, such as individual parameters having to have the same values or resulting from a combination of other parameters. The parameter lists can then be linked to the abstracted test module a1, a2, a3, a4 generated from the respective test module T11, T12, T13, T14. For this purpose, for example, the respective parameter list can be saved in the test system or in the associated database with the respective abstracted test module a1, a2, a3, a4.
[0048] Subsequently, in a synthesis step 103, abstract test scenarios TS1, TS2, TS3 are created for the security check of the respective functional system to be tested by combining the abstracted test modules a1, a2, a3, a4. Abstracted test modules a1, a2, a3, a4, a5, which were generated from various test cases TC1—for example, already applied and / or applicable—can be combined to form new test scenarios TS1, TS2, TS3 for the security check of the functional system to be tested. The created test scenarios TS1, TS2, TS3, or at least parts of the created test scenarios TS1, TS2, TS3, TS4 can also be saved in the test system or in the associated database, for example for reuse for similar or the same test requirements. For this purpose, the corresponding description language (e.g.Domain Specific Language) can be used.
[0049] Furthermore, in synthesis step 103, each newly created test scenario TS1, TS2, TS3 can be searched for the required parameters which are to be determined or provided by the functional system to be tested and / or at least one of the preceding test modules T21, T22, T23, T24, T25 when executing a test case TC21, TC22, TC23 derived from the test scenario TS1, TS2, TS3. In this way, for example, ineffective, non-functional and / or meaningless test scenarios TS1, TS2, TS3 (e.g., due to the order of the abstracted test modules a1, a2, a3, a4, a5) can be identified and eliminated as early as synthesis step 103. In order to be able to recognize and locate these required parameters more quickly in the respective test scenario TS1, TS2, TS3, the parameter lists created in the abstraction step 102 can be used, for example.For this purpose, the parameter lists are used which are linked to the abstracted test modules a1, a2, a3, a4, a5 used in the test scenario TS1, TS2, TS3.
[0050] In a subsequent concretization step 104, concrete test cases TC21, TC22, TC23 for the security check of the functional system to be tested are then derived from the abstract test scenarios TS1, TS2, TS3. For this purpose, the respective abstract placeholder variables in the abstracted test modules a1, a2, a3, a4, a5 of the test scenarios TS1, TS2, TS3 are assigned information, data, and parameters of the functional system to be tested. The system-specific information, data, and parameters of the system currently to be tested can be provided, for example, by a system manufacturer or by a client of the security check to be performed. Alternatively or additionally, empirical values from previous security checks of the same or a similar functional system can also be used. This empirical value can, for example, also be stored in the test system or in the associated database.Alternatively or additionally, the system-specific information, data and parameters can originate at least partially from a machine learning system instead of from the database, which generates the information, data and parameters based on empirical values. Furthermore, system-specific information, data and parameters of the system to be tested, with which the abstract placeholder variables in the abstracted test modules a1, a2, a3, a4, a5 of the test scenarios TS1, TS2, TS3 can be assigned, can also be determined from an analysis of the functional system currently being tested (e.g. scans on the system, port scan, etc.) or from available manufacturer data. After the concretization step 104, the concrete test cases TC21, TC22, TC23 derived from the abstract test scenarios TS1, TS2, TS3 are applied to the functional system to be tested in an application step 105.In application step 105, for example, during the ongoing security check, it can be documented which concrete test cases TC21, TC22, TC23 are particularly well suited for uncovering vulnerabilities, which test cases TC21, TC22, TC23 deliver the best result, which test cases TC21, TC22, TC23 do not deliver particularly good results, which test cases TC21, TC22, TC23 do not function or only function incorrectly on the functional system to be tested, etc.
[0051] After the application step 105, information collected during the execution of the security check on the respective test cases TC21, TC22, TC23 can then be transmitted to the test system in a feedback step 106. For this purpose, for example, a feedback can be created for each test case TC21, TC22, TC23, which contains, for example, information on the execution of the respective test case TC21, TC22, TC23. The feedback can then be transmitted to the test system and linked there, for example, with the test scenarios TS1, TS2, TS3 from which the test cases TC21, TC22, TC23 were derived. The feedback can then be used to identify test cases TC21, TC22, TC23 that function particularly well or test cases TC21, TC22, TC23 that do not function or only function incorrectly.Furthermore, for each abstract test scenario TS1, TS2, TS3, information can be stored regarding the functional system for which the respective test scenario TS1, TS2, TS3 has already been applied. This allows test scenarios TS1, TS2, TS3 that have already been tested and found to be successful or well-functioning to be identified in the synthesis step 103 during a subsequent security check of the same or a similar functional system in the test system and used again in the concretization step 104 to generate test cases TC21, TC22, TC23. Furthermore, test scenarios TS1, TS2, TS3 that do not function, are unsuitable for a specific functional system, or are pointless (e.g., due to the test module sequence, cannot be executed on a functional system due to required parameters, etc.) to be very easily eliminated from the test system or the associated database.The test system can thus be continuously improved and learn from security checks already carried out - in particular from concretization steps 104.
[0052] Figures 2a to 2c describe the abstraction step 102, the synthesis step 103, and the concretization step 104 of the method in more detail using an exemplary embodiment. In the exemplary embodiment, for example, a security check is to be performed for a specific vehicle, for which, for example, a security check is to be carried out after a new development of one or more components or after an update of one or more components. The aim is to identify, for example, vulnerabilities or points of attack for cyberattacks on this vehicle.
[0053] Figure 2a shows an example of the abstraction step 102. This is based, for example, on an exemplary concrete test case TC1 which has already been applied, for example, to a similar functional system (e.g., a vehicle from another manufacturer, etc.) or to the same functional system (e.g., a vehicle before a component replacement or update). Test case TC1 represents, for example, a concrete attack vector on a functional system that has already been subjected to a security check. With this concrete attack vector, an attempt was made, for example, to reach the engine control of the functional system or vehicle via an attack on an external interface (e.g., Bluetooth, radio, etc.) of the functional system.
[0054] The exemplary test case TC1 can, for example, be divided into four test modules T11, T12, T13, T14 in the modularization step 101. In this case, each test module T11, T12, T13, T14 corresponds, for example, to a specific attack on a component or on a subsystem of the functional system or vehicle under test. The first test module T11 can, for example, represent an attack on an external interface of the vehicle, with which, for example, existing external interfaces (e.g. wireless communication interface, sensors, etc.) are located. A second test module T12 attempts, after a successful attack using the first test module T11, to access the vehicle's infotainment system, for example. If the second test module T12 is successfully executed, a third test module T13 attempts, for example, to attack or use the vehicle's internal network.If the third test module T13 is successful, an attempt is made, for example, in a fourth test module T14 to take over the vehicle's engine control.
[0055] In the abstraction step 102, the concrete test modules T11, T12, T13, T14, into which the concrete test case TC1 was divided, are now abstracted. This means that in the test system, all system-specific information, data, and parameters of the vehicle tested with the test case TC1 are removed from the test modules T11, T12, T13, T14 and replaced with abstract placeholder variables. The test modules T11, T12, T13, T14 are thus abstracted test modules a1, a2, a3, a4, which abstractly describe the respective attacks. For example, an abstracted test module a1 derived from the first test module T11 can contain the description to search for a wireless interface of (any) functional system and to establish a connection with it. A second abstracted test module a2 derived from the second test module T12 can, for example,contain the description of how, after a successful connection to the wireless interface, a connection is to be established with a component or subsystem of the functional system that can be reached via the found interface. In a third abstracted test module a3 derived from the third test module T13, for example, the use of an internal network of the functional system is described abstractly. A fourth abstracted test module a4 can, for example, abstractly describe the takeover of control of the functional system. The abstracted test modules a1, a2, a3, a4 can then be stored in the test system or in the associated database, for example, provided that these abstracted test modules a1, a2, a3, a4 are not already stored in the test system or in the associated database.
[0056] Furthermore, the test modules T11, T12, T13, and T14 can be searched for required parameters before abstraction, and test-module-specific parameter lists can be created accordingly. The respective test-module-specific parameter list can be linked to the respective abstracted test module a1, a2, a3, and a4 and stored with it in the test system or in the associated database.
[0057] Figure 2b illustrates synthesis step 103 of the method based on an exemplary embodiment, such as a safety check of a vehicle currently being tested. In this case, the abstracted test modules a1, a2, a3, a4, a5 are combined to form abstract test scenarios TS1, TS2, TS3, with each test scenario TS1, TS2, TS3 representing an abstract description of a possible attack vector. To create the test scenarios TS1, TS2, TS3, for example, the abstracted test modules a1, a2, a3, a4 derived from the concrete test case TC1 in abstraction step 102, as well as further abstracted test modules a5, which were abstracted from test cases TC1 of other, already performed safety checks, can be used. Furthermore, abstracted test modules a5 can also be used, which are already stored in the test system or in the associated database.
[0058] In the synthesis step 103 shown in Figure 2b, for example, the abstracted test modules a1, a2, a3, a4 generated in the abstraction step 102 shown in Figure 2a are combined to form a first test scenario TS1. In a further, second test scenario TS2, for example, the second, third and fourth abstracted test modules a2, a3, a4 from the concrete test case TC1 are combined with a fifth abstracted test module a5 (e.g., an abstract description that a connection to a system-internal component (e.g., gateway) is to be established and / or used), wherein the fifth abstracted test module a5 is inserted, for example, between the third and fourth abstracted test modules a3, a4.In a further, third test scenario TS3, the second, third and fourth abstracted test modules a2, a3, a4 are also combined with the fifth abstracted test module a5, whereby in the third test scenario the fifth abstracted test module a5 is arranged, for example, between the second and the third abstracted test modules a2, a3.
[0059] In synthesis step 103, abstract test scenarios TS1, TS2, TS3 can be created from the test system using any combination of the abstracted test modules a1, a2, a3, a4, a5. In order to exclude, for example, ineffective or pointless combinations of abstracted test modules a1, a2, a3, a4, a5 - such as a combination in which, for example, the fourth abstracted test module a4 (e.g., taking over control of the functional system) is placed before the first test module a1 (e.g., searching for external interfaces of the functional system as a point of attack), the created abstract test scenarios can be searched for required parameters, or the parameter lists stored for the respective abstracted test modules a1, a2, a3, a4, a5 can be evaluated. Furthermore, to create abstract test scenarios TS1, TS2, TS3 in the test system orSuggestions for combinations of abstracted test modules a1, a2, a3, a4, a5 available in the associated database can be used to create test scenarios TS1, TS2, TS3. Furthermore, the created abstract test scenarios TS1, TS2, TS3, or at least parts thereof, can be stored in the test system or in the associated database in synthesis step 103—for example, as suggestions for creating test scenarios TS1, TS2, TS3 in subsequent synthesis steps 103.
[0060] Figure 2c shows the concretization step 104 of the method using the concrete embodiment - e.g., the safety check of a vehicle currently being tested. In the concretization step 104, concrete test cases TC21, TC22, TC23 for the functional system currently being tested - e.g., a vehicle with at least one newly developed or updated component - are derived from the abstract test scenarios TS1, TS2, TS3 created in the synthesis step 103. The abstract placeholder variables in the abstracted test modules a1, a2, a3, a4, a5 of the respective test scenarios TS1, TS2, TS3 are assigned system-specific information, data, and parameters of the functional system or vehicle currently being tested. The first test scenario TS1 thus becomes a concrete, first test case TC21 for the safety check of the vehicle currently being tested. Furthermore, the second test scenario TS2 orthe third test scenario TS3 for a specific, second test case TC22 or for a specific, third test case TC23. The system-specific information, data, and parameters can originate from the client of the safety check or from the vehicle's manufacturer data. For example, empirical values from previous safety checks of the vehicle or similar vehicles can be used, or data from analyses, etc., of the vehicle currently being tested can be used. The test cases TC21, TC22, TC23 derived in the concretization step 104 are then used in the application step 105 for the safety check of the functional system or vehicle currently being tested. The findings and experience from the application of the test cases TC21, TC22, TC23 can then be transmitted to the test system in the feedback step 106 in the form of feedback on the test cases TC21, TC22, TC23.There they can be linked and stored with the corresponding test scenarios TS1, TS2, TS3 in order to be able to use the feedback again in subsequent safety checks of functional systems.
Claims
Patent claims 1 . A computer-implemented method for adapting test cases (TC1) for a security check of a functional system (105) of a mobility application to be tested by means of a test system, wherein test cases (TC1) are made available to the test system which have been used in security checks of functional systems used in mobility applications and / or are applicable for such security checks, and wherein the following steps are carried out: - Dividing the test cases (TC1) into test modules (T 11 , ... , T14) (101); - abstracting the test modules (T 11 , ... , T14), whereby system-specific information, data and parameters of the functional systems to which the respective test cases (TC1) were applied, contained in the test modules (T 11 , ... , T14), are replaced by abstract placeholder variables in a respective test module (T 11 , ... , T14) (102); - Creation of abstract test scenarios (TS1, TS2, TS3) for the safety testing of the functional system to be tested, whereby the abstracted test modules (a1, ..., a5) are combined for a test scenario (TS1, TS2, TS3) (103); - Deriving test cases (TC21, TC22, TC23) for the functional system to be tested from the abstract test scenarios (TS1, TS2, TS3), whereby the respective abstract placeholder variables in the abstracted test modules (a1, ..., a5) of a respective test scenario (TS1, TS2, TS3) are assigned corresponding system-specific information, data and parameters of the functional system to be tested (104); and - Applying the test cases (TC21, TC22, TC23) derived from the test scenarios (TS1, TS2, TS3) to the functional system to be tested (105).
2. Method according to claim 1, characterized in that after applying the test cases (TC21, TC22, TC23) derived from the test scenarios (TS1, TS2, TS3) to the functional system to be tested, a feedback is created for each test case (TC21, TC22, TC23), that the feedback is transmitted to the test system and is linked (106) in the test system with the respective test scenario (TS1, TS2, TS3) from which the respective test case (TC21, TC22, TC23) was derived.
3. Method according to one of the preceding claims, characterized in that during abstraction the test modules (T 11 , ... , T14) are searched (102) for required parameters which are to be made available for execution of the respective test module (T 11 , ... , T14) by the functional system to be tested (105) or at least one of the preceding test modules (T11 , ... , T14), and that the required parameters are stored in a parameter list in the test system, wherein the the respective parameter list is linked to the test module (a1, a5) abstracted from the respective test module (T11, T14).
4. Method according to one of the preceding claims, characterized in that a created, abstract test scenario (TS1, TS2, TS3) is searched (103) for placeholder variables of required parameters which are to be provided to the respective test module (T21, ..., T25) by the functional system to be tested and / or by at least one of the preceding test modules (T21, ..., T25) for carrying out the test case (TC21, TC22, TC23) derived from the test scenario (TS1, TS2, TS3).
5. The method according to claim 4, characterized in that in order to find the placeholder variables of the required parameters in the respective test scenario (TS1, TS2, TS3), the parameter lists linked to the respective abstracted test modules (a1, ..., a5) of the respective test scenario (TS1, TS2, TS3) are used (103).
6. Method according to one of the preceding claims, characterized in that the abstracted test modules (a1, ..., a5) and / or abstract test scenarios (TS1, TS2, TS3) and / or parts of abstract test scenarios (TS1, TS2, TS3) are stored in the test system, in particular in a database (102, 103).
7. The method according to claim 6, characterized in that before storing the respective abstracted test module (a1, ..., a5), it is checked (102) whether the respective abstracted test module (a1, ..., a5) is already stored in the test system, in particular in the database.
8. Method according to one of the preceding claims, characterized in that for each abstract test scenario (TS1, TS2, TS3) in the test system, information is stored (103) as to which functional systems the respective test scenario (TS1, TS2, TS3) has been applied.
9. Method according to one of the preceding claims, characterized in that the system-specific information, data and parameters of the functional system to be tested for occupying the placeholder variables in the abstracted test modules (a1, ..., a5) of the respective test scenarios (TS1, TS2, TS3) are provided by the test system, in particular by the database in which empirical values from previous security checks of an identical or similar functional system are stored, and / or by a system manufacturer and / or are determined by means of machine learning from empirical values from previous security checks (104).
10. The method according to one of claims 1 to 9, characterized in that values which are determined from an analysis of the functional system to be tested are used (104) as system-specific information, data and parameters of the functional system to be tested for filling the placeholder variables in the abstracted test modules (a1, a5) of the respective test scenarios (TS1, TS2, TS3).
11. Test system with at least one computer unit which is configured to carry out the method according to one of claims 1 to 10.
12. Test system according to claim 11, wherein the test system further comprises at least one storage unit and / or is connected to a storage unit which is configured to store abstracted test modules (a1, ..., a4, a5), abstract test scenarios (TS1, TS2, TS3) and / or parts of abstract test scenarios (TS1, TS2, TS3) as well as empirical values from previous security checks of the same or similar functional system.
13. A computer program product, wherein the computer program product can be loaded into the test system according to one of claims 11 to 12, and wherein the computer program product comprises instructions which, when executed by the test system, cause the computer program product to carry out the method according to one of claims 1 to 10.