System and method for automated vulnerability scanning

Abstract

To provide a system and method for automated vulnerability scanning.SOLUTION: A system and method for testing may include automatically initiating a test procedure, the procedure including: causing at least one message to be transmitted over a network which is internal to a tested system; receiving data communicated by a component in the tested system; identifying, based on the received data, a deviation from at least one of an expected behavior of at least one component of the tested system and a protocol or a communication standard used by the tested system; and automatically associating a specific test with the deviation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates generally to automated vulnerability scanning, detection, and identification. [Background technology]

[0002] Modern automotive systems are complex environments consisting of multiple components, subsystems, and various communication channels that allow these subsystems and components to communicate with each other as well as with external systems. Typically, each subsystem is made up of or includes various hardware and software components and can send and / or receive information using various communication channels and protocols, such as to send or receive sensor data, commands, reports, or to control other subsystems or components.

[0003] Systems, subsystems, and components contained within vehicles are known to be inherently prone to defects (also referred to in the art as vulnerabilities) that can be exploited by malicious entities (e.g., attackers) to externally manipulate the control of or disrupt the function of the vehicle's systems and components. To reduce the risk of such vulnerabilities, various techniques are used to detect and eliminate them during the development process. Generally, testing a system for vulnerabilities is referred to in the art as scanning; for example, testing a system to discover, detect, or identify vulnerabilities is referred to in the art as a vulnerability scan or scanning process.

[0004] The ability to interface, interact, and communicate with the system under test during a vulnerability scan will affect the time it takes to complete the scan and the effectiveness of the scan process, i.e., whether all, or at least most, vulnerabilities are detected, identified, or discovered during the scan.

[0005] However, known vulnerability scanning systems and methods require extensive scan time to complete, provide insufficient coverage of the system under test, and typically require a human user to monitor, execute, and control the scanning process. Furthermore, known current vulnerability scanning systems and methods require detailed knowledge and understanding of the system under test in order to know which specific tests to run (e.g., depending on the specific components of the system under test), what the expected results / outcomes of those tests are, and what the test results actually mean. To overcome some of these issues, known current vulnerability scanning systems and methods include incorporating agents into the system under test. Such agents are used to internally monitor and report on the operation of the system under test. Obviously, inserting, planting, or including an agent is a time-consuming and labor-intensive task, e.g., specific agents must be created for each specific system, new systems (including the agents) must be created, and the use of agents also has other downsides, e.g., agents may interfere with the intended functionality of the system under test. Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, a solution to the above-mentioned drawbacks may be sought. This object is achieved by the subject matter of the independent claims. The dependent claims disclose optional embodiments thereof. [Means for solving the problem]

[0007] One embodiment of the test includes automatically initiating a test procedure for a system under test, the procedure including causing at least one message to be transmitted over a network internal to the system under test, receiving data communicated by a component within the system under test, identifying a deviation from expected operation of at least one component of the system under test and at least one of a communication protocol or standard used by the system under test based on the received data, and automatically associating a specific test with the deviation.

[0008] Some embodiments include generating traffic on the internal network by selectively randomizing data in messages transmitted on the internal network. At least one service provided by at least one component of the system under test is automatically identified based on the received data. In some embodiments, the types and / or functions of components included within the system under test are automatically identified and / or specified. The test procedure is automatically modified based on at least one of the services and components discovered within the system under test.

[0009] In some embodiments, a particular component is automatically associated with a deviation. In some embodiments, the state of a component is determined based on a response or absence of a response to a message sent within the system under test. Deviations may be identified based on the absence of a message, unexpected content within a message, the absence of a change to a message, and a discrepancy with an expected sequence of events. An identified deviation may be a crash or restart of a component or service.

[0010] In one embodiment, a list of components included in the system under test is received, each of the components in the list is associated with a component automatically identified on the network based on a message from the component, and if any of the components in the list are not associated with a component on the network, the embodiment updates a log and / or alerts a user.

[0011] In one embodiment, the message is automatically generated based on at least one of automatically identified services provided by components in the system under test and types and functions of components included in the system under test. In one embodiment, the message is generated based on at least one of protocols conforming to the components in the system under test, services provided by the components in the system under test, and functions of the components in the system under test.

[0012] In some embodiments, the procedure or test that caused the deviation from expected behavior is identified. In some embodiments, the system under test is designed / adapted to be installed in a vehicle and the network internal to the system under test is an in-vehicle network. In some embodiments, the deviation includes one of a malfunctioning hardware component and a malfunctioning software component. In some embodiments, the set of tests to be included in the test procedure is automatically selected based on the number and type of components of the system under test and the services provided thereby.

[0013] In some embodiments, the test procedure includes a plurality of tests, and the method includes automatically determining whether the system under test passed at least one of the tests. In some embodiments, the test procedure includes a plurality of tests, and the method includes automatically determining a state of the system under test after performing at least one of the tests. In some embodiments, the message is automatically generated and transmitted according to at least one of a protocol, procedure, component, and / or structure of the system under test. Other aspects and / or advantages of the present invention are described herein.

[0014] Non-limiting examples of embodiments of the present disclosure are described below with reference to the accompanying drawings hereof, listed following this paragraph. Identical features shown in multiple drawings are typically labeled with the same reference numeral in all drawings in which they appear. Reference numerals representing icons representing certain features of embodiments of the present disclosure may be used to refer to that feature. Dimensions of features shown in the figures have been chosen for convenience and clarity of presentation and are not necessarily drawn to scale. For example, for clarity, the dimensions of some of the elements may be exaggerated relative to other elements, or several physical components may be included in a single functional block or element. Furthermore, where considered appropriate, reference numerals may be repeated between views to indicate corresponding or similar elements.

[0015] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. The invention, however, both as to organization and method of operation, together with its objects, features, and advantages, may best be understood by reference to the following detailed description taken in conjunction with the accompanying drawings. Embodiments of the invention are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like reference numerals indicate corresponding, similar, or similar elements. [Brief explanation of the drawings]

[0016] [Figure 1]1 illustrates a block diagram of a computing device in accordance with an exemplary embodiment of the present invention. [Figure 2] 1 illustrates a system according to an exemplary embodiment of the present invention. [Figure 3] 1 illustrates a system according to an exemplary embodiment of the present invention. [Figure 4] 1 shows a flowchart of a method according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] As previously discussed, current systems and methods for vulnerability scanning take a long time to complete a full or substantial scan, typically do not provide adequate coverage of the system under test, and typically require a human user to be involved in the scanning process.

[0018] Embodiments of the present invention provide and enable new methods for automatically detecting, revealing, and / or identifying vulnerabilities and their impact on systems under test, thereby enabling more vulnerabilities to be detected and addressed before systems are sold to consumers.

[0019] In general, embodiments of the present invention include a computing device, such as computing device 100 as further described herein, or embodiments include only a central processing unit (e.g., controller 105 as further described herein) that can generate either random data or specific sequences for predetermined protocols and use cases known to be critical to security functionality.

[0020] Additionally, some embodiments of the present invention include network or other interfaces that allow communication with the system under test using different network architectures and protocols, such as Controller Area Network (CAN), Ethernet, Wi-Fi (WiFi), etc.

[0021] Other protocols that may be applicable and used as described herein with reference to a "protocol" or "particular protocol" may be Scalable service-Oriented Middleware over IP (SOME / IP), Universal Measurement and Calibration (XCP), Diagnostics over Internet Protocol (DoIP), Unified Diagnostic Services (UDS).

[0022] To automate the scanning process and increase its effectiveness, some embodiments include and / or use multiple mechanisms that can directly indicate the impact of specific communication patterns on the system under test. Such mechanisms dramatically accelerate the speed at which different scenarios can be tested. By being able to investigate more scenarios in less time, embodiments of the present invention significantly reduce the complexity of the process of covering complex systems. Additionally, the mechanisms for generating indicators as described allow pinpointing the exact scenario in which the undesired impact occurs, allowing operators to quickly understand the system functionality that failed, quickly perform a defect or vulnerability analysis, and propose a solution.

[0023] Unlike current and / or known testing systems and methods, embodiments of the present invention can perform true, actual, or real-world black-box testing of a system. For example, using the mechanisms described herein, embodiments of the present invention can connect to a system under test without any prior knowledge of the system under test, perform a testing procedure including a series of tests or tests, and provide indicators and reports that clearly explain, for example, which tests were passed or failed, which components failed during which tests, what vulnerabilities were identified, etc. Furthermore, it should be noted that black-box testing according to embodiments of the present invention as described above can be performed without any additions or modifications to the system under test; for example, the system under test can be tested in the same configuration, capacity, or configuration as intended for use during production, e.g., on a vehicle.

[0024] 1, which illustrates a non-limiting block diagram of a computing device or system 100 that may be used by or included in a system according to some embodiments of the present invention. For example, computing device 100 may be an ECU onboard a vehicle, such as ECU 250, or a sensor 251 as described further herein. For clarity and brevity, the term ECU is used herein, but it should be understood that any related component or element may be applicable, such as an IoT device, a smart sensor, etc.

[0025] In some embodiments, computing device (or system) 100 includes controller 105, which is a hardware controller. For example, computer hardware processor or hardware controller 105 can be or include a central processing unit processor (CPU), chip, or any suitable computing or computational device. In some embodiments, computing system 100 includes memory 120, operating system (OS) 115, executable code 125, storage system 130, communication infrastructure 150, input / output (I / O) modules 140, and I / O components 135, which may be removably connected to computing device 100. In some embodiments, controller 105 (or one or more controllers or processors, possibly across multiple units or devices) is configured to perform methods described herein (e.g., by executing software or code) and / or to perform or function as various modules, units, etc., e.g., by executing software or using dedicated circuitry. More than one computing device 100 may be included in a system according to some embodiments of the present invention, and more than one computing device 100 may be or function as a component of a system according to some embodiments of the present invention.

[0026] In some embodiments, OS 115 may include any code segments (e.g., similar to executable code 125 described herein) designed and / or configured to perform tasks including coordinating, scheduling, arbitrating, monitoring, controlling, or otherwise managing the operation of computing device 100, such as scheduling the execution of software programs or enabling communication between software programs or other modules or units. For example, OS 115 may be a commercial operating system, such as QNX or Linux.

[0027] In some embodiments, memory 120 is hardware memory. For example, memory 120 may be or include a machine-readable medium for storing software, such as random access memory (RAM), read-only memory (ROM), memory chips, flash memory (referred to herein as flash), volatile and / or non-volatile memory, or other suitable memory or storage unit. Memory 120 may be or include multiple, possibly different, memory units. In some embodiments, memory 120 is a computer or processor non-transitory readable medium or a computer non-transitory storage medium, such as RAM. Some embodiments may include a non-transitory storage medium having stored thereon instructions that, when executed, cause a processor to perform the methods disclosed herein.

[0028] Executable code 125 may be software, an application, a program, a process, a task, or a script. A program, process, task, application, or software referred to herein may be any type of instruction, such as firmware, middleware, microcode, hardware description language, etc., that, when executed by one or more hardware processors or controllers 105, causes a processing system or device (e.g., system 100) to perform various functions described herein. In some embodiments, executable code 125 is executed by controller 105, possibly under the control of operating system 115.

[0029] For example, executable code 125 may be an application that automatically performs vulnerability scans, as described further herein. Although a single item of executable code 125 is shown in FIG. 1 for clarity, systems according to some embodiments of the present invention may include multiple executable code segments similar to executable code 125 that may be loaded into memory 120 and cause controller 105 to perform methods described herein.

[0030] Executable code 125 may be or may include several applications, services, routines, or other software modules or components. It should be understood that components of the systems described herein (e.g., components of ECU 230 and / or system under test 260) may be software components, such as applications, services, routines, or other software components. For example, controller 105 in test unit 210 may send a message to a first software service provided or executed by controller 105 in ECU 230 based on a message sent by / received from a second, different software service provided or executed by controller 105 in ECU 230. For clarity and brevity, controller 105 in test unit 210 will be referred to herein simply as test unit 210, and where applicable, "test unit 210" should be understood to mean "controller 105 in test unit 210."

[0031] Storage system 130 may be or include, for example, a hard disk drive (e.g., when connected to test unit 210 as further described), a universal serial bus (USB) device, a flash memory or unit (e.g., in 250), or any other suitable removable and / or non-removable storage unit. In some embodiments, some of the components shown in FIG. 1 may be omitted. For example, in an ECU, storage system 130 may be non-volatile memory (e.g., flash memory) included in memory 120 of the ECU. Thus, although storage system 130 is illustrated as a separate component, storage system 130 may be embedded in or included in system 100, e.g., memory 120.

[0032] Computing device 100 may include communications infrastructure 150, which may be, for example, a bus, such as a CAN or other bus, or other system connected to controller 105, operating system 115, memory 120, and I / O modules 140, allowing controller 105, operating system 115, memory 120, and I / O modules 140 to communicate. I / O modules 140 may include ports or other hardware outlets that allow external devices to communicate with internal components; for example, I / O modules 140 may allow external devices to write data to memory 120, send commands to controller 105, etc.

[0033] I / O components 135 may be used to connect (e.g., via included ports) or they may include a mouse, keyboard, touchscreen or pad, or any other suitable input device. I / O components may include one or more screens, touchscreens, displays or monitors, speakers, and / or any other suitable output device. Any applicable I / O component may be connected to computing device 100 as represented by I / O component 135; for example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device, or an external hard drive may be included in I / O component 135. For simplicity, storage system 130 is shown as connected to communications infrastructure 150, although storage system 130 may be connected via I / O module 140, e.g., a port therein.

[0034] Systems according to some embodiments of the present invention may include components such as, but not limited to, multiple central processing units (CPUs) or any other suitable general-purpose or special-purpose processors, controllers, microprocessors, microcontrollers, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), application-specific integrated circuits (ASICs), etc. Systems according to some embodiments of the present invention may include multiple input units, multiple output units, multiple memory units (e.g., memory 120 in an ECU may include flash memory and RAM), and multiple storage units. The system may additionally include other suitable hardware and / or software components. In some embodiments, the system may include or be, for example, a server computer, a network device, and any number of devices, such as Internet of Things (IoT) devices, automotive ECUs, e.g., ECUs installed in cars, trains, ships, or airplanes.

[0035] Referring to FIG. 2, a system 200 according to an exemplary embodiment of the present invention is shown. As shown in FIG. 2, the system may include a test unit 210, which may include or be operatively connected to a storage unit or system 130. As further shown, the storage unit 130 may include one or more tests 270 and one or more test procedures 271. As shown, the system 200 may include a system under test 260, a network 240, and connecting lines 280 that enable communication between the test unit 210 and the system under test 260. At least some of the components and elements shown in FIG. 2 as being included in the system 200 are optional. For example, the network 240 that may enable communication between the test unit 210 and the system under test 260 is an optional element and may be removed from or not present in the system 200 (e.g., as shown in FIG. 3). For example, communication between the test unit 210 and the system under test 260 may occur only over the connecting lines 280 in a system 200 that does not include the network 240. The storage unit or system 130 is described with respect to FIG.

[0036] Test unit 210 may be device 100 or may include components of device 100. For example, test unit 210 may be a computer or server known in the art. Test 270 may be an electronic file or other digital file or entity that stores information. For example, test 270 may define, list, describe, or include any information necessary to dynamically send messages over a communication line, bus, or network internal to system under test 260. For example, rules, parameters, or values ​​that define or describe aspects such as the source or destination addresses to be used in the message, the content of the message, how often the message is sent, the number of times it is sent, etc. may all be defined in test 270. Test 270 may define or include the logic or flow of the test. For example, logic within test 270 may select a response message to send upon receipt of a particular message from an internal bus of system under test 260.

[0037] Tests 270 may generally be designed to cover various aspects, operations, or components of system under test 260. For example, a first test 270 may be directed to an infotainment system if system under test 260 is designed for a vehicle, e.g., if the test involves sending messages intended for the infotainment system, and a second test 270 may be directed to a headlight controller.

[0038] Test 270 may define, list, describe, or include any information of data necessary to identify deviations from expected operation of at least one component of system under test 260 based on data received from system under test 260. Test 270 may define, list, describe, or include any information of data necessary to identify deviations from a protocol or communication standard used by the system under test based on data received from system under test 260.

[0039] For example, the test unit 210, which has detailed knowledge of protocols such as SOME / IP, XCP, DoIP, and UDS, tracks the flow of messages and identifies or determines whether this flow complies with a particular protocol, i.e., the test unit 210 identifies protocol violations or deviations from the protocol.

[0040] For clarity and brevity, the terms "deviation from expected behavior" and "deviation from expected protocol" may also be referred to herein simply as "deviation," and these terms may be used interchangeably herein. The terms "deviation," "fault," and "failure," as referred to herein, may mean the same thing and may be used interchangeably herein.

[0041] For example, using the information in test 270, test unit 210 can determine whether system under test 260 malfunctioned, crashed, rebooted, or otherwise failed the test. Using the information in test 270, test unit 210 can generate or select one or more messages to send over internal communication infrastructure 253 in response to messages received from internal communication infrastructure 253. Test unit 210 can recognize the current test 270 and record any events related to the current test 270. For example, when test unit 210 identifies a crash, reboot, malfunction, or other unexpected behavior of the system under test, it can record the current test in association with any information related to the crash or malfunction, such as which particular ECU 250 or service within ECU 230 was being tested when the crash occurred. Tests 270 may include human-readable material describing the particular components the test 270 is designed for and covers, so that by simply pointing to a particular test 270, the test unit 210 automatically provides useful information to the user.

[0042] The system under test 260 may include one or more ECUs 250, one or more sensors 251, a communication infrastructure 253, which may be, for example, a bus, CAN, or Ethernet network infrastructure. As further shown, the system under test 260 may include flash memory 223 and random access memory (RAM) 222.

[0043] System under test 260 can be any device, system, or group of components intended to be installed, for example, in a vehicle. For example, system under test 260 or parts or components of system under test 260 can be, be included in, or be integrated into an IoT device or a vehicle, such as an airplane, car, train, or watercraft. While vehicles such as automobiles are primarily described herein, other vehicles, such as watercraft, trains, and aircraft, are equally applicable, and it should be understood that the scope of the present invention is not limited by the type, use, or nature of system under test 260. As referred to herein, components of the system under test (e.g., system under test 260 and / or ECU 230) can be software components, such as applications, services, routines, or other software modules or components.

[0044] ECU 250, sensor 251, and test procedure 271 may hereinafter be referred to collectively as ECU 250, test procedure 271, and sensor 251, or individually as ECU 250, test procedure 271, and sensor 251, merely for the sake of brevity.

[0045] Sensor 251 can include more than just a temperature or pressure sensing or measuring unit; for example, sensor 251 can include computing device 100 or portions thereof; for example, sensor 251 can include memory 120, controller 105, and / or executable code 125; sensor 251 can store and process measurements, receive network packets, and transmit network packets, messages, or frames. For example, sensor 251 can be an oil pressure sensor, a temperature sensor, etc., e.g., if an embodiment is or is included in a train, car, or other vehicle. If an embodiment is included in an airplane, watercraft, or spacecraft, sensor 251 can be a unit that senses or measures, e.g., altitude, acceleration, airspeed, or current.

[0046] ECU 250 may be computing device 100 or may include any component thereof. For example, ECU 250 may include controller 105, memory 120 (e.g., RAM and flash memory), may include communication infrastructure 150 in the form of a bus, and may include I / O modules 140. Typically, ECU 250 is associated with or designed for a particular component; for example, a first ECU 250 may monitor and control the engine in an automobile, and a second ECU 250 may monitor and control the windows. When an embodiment is included in an airplane, watercraft, or spacecraft, ECU 250 may be a Line-Replaceable Unit (LRU), Lower Line-Replaceable Unit (LLRU), Line-Replaceable Component (LRC), or Line-Replaceable Item (LRI) component.

[0047] Communication infrastructure 253 may be any communication line, bus, or network, such as, for example, an Ethernet network, a CAN bus, etc. For clarity, communication infrastructure 253 is referred to herein as network 253, and communication infrastructure 150 is referred to herein as bus 150. Memory 223 may be a component of or include memory 120 as described herein. RAM 222 may be used by controller 105 in the system under test to execute program applications, for example, as known in the art. Communication infrastructure 253 may be any suitable network that allows ECU 250 and sensors 251 to communicate or exchange digital information, as well as read / write data from / to memories 222 and 223. Communications infrastructure 253 may be, encompass, or include any suitable bus or network infrastructure and may support any suitable protocol; for example, in-vehicle network 253 may be an Ethernet network, a CAN network, or a combination thereof, and protocols such as Internet Protocol (IP), Transmission Control Protocol (TCP), etc. may be used to transmit data over communications infrastructure 253. Accordingly, many elements of communications infrastructure 253 are not shown but are implied, such as switches, routers, gateways, etc. It should be understood that embodiments of the present invention are not limited by the nature of communications infrastructure 253.

[0048] Network 240 can be any suitable network that enables test unit 210 to communicate with system under test 260. For example, network 240 can be a private or public IP network, the Internet, or a combination thereof. Network 240 can also be a custom bus or network that includes hardware wires that directly connect test unit 210 to system under test 260. Thus, many elements of network 240 are implied but not shown. For example, network 240 can be or include a Wi-Fi network, and messages exchanged as described above between test unit 210 and system under test can be transmitted over the Wi-Fi network within network 240.

[0049] Connection line 280 may be or include any hardware and software necessary to directly connect test unit 210 with one or more ports of system under test 260. For example, connection line 280 may be a wire that connects to test unit 210 and has a jack that mates with a jack in I / O module 140 of system under test 260. For example, connection line 280 may plug into a port on system under test 260, such that all data communications over communications infrastructure 253 are received by test unit 210. Connection line 280 may plug into a port on system under test 260, such that test unit 210 can transmit messages over communications infrastructure 253.

[0050] 3, which illustrates a system 300 according to an exemplary embodiment of the present invention. ECU 230 may be similar to or identical to ECU 250 described herein, and reference number 230 is used for clarity. Connection line 290 may be similar to or identical to connection line 280 or multiple connection lines 280 described herein, and reference number 290 is used for clarity.

[0051] 3 , ECU 230 may be used as or serve as system under test 260. Any systems or operations described herein with respect to system under test 260 may be applicable here. For example, test unit 210 may receive messages sent over communication infrastructure 150, which may be internal to ECU 230, and test unit 210 may send messages over communication infrastructure 150. In some embodiments, test unit 210 is connected to an internal communication infrastructure (e.g., communication infrastructure 150 or communication infrastructure 253) within the system under test (e.g., system under test 260 or ECU 230) over one single interface or line, such as a network or bus interface, and automatically performs the automatic test procedures described herein based on messages received from the internal communication infrastructure. Stated another way, the test unit 210 can perform all of the methods described herein if the only input it receives from a test system, for example from the system under test 260 or from the ECU 230, is a message sent over a communication bus, network, or other infrastructure internal to the system under test.

[0052] As shown in FIG. 3 , the test unit 210 may be connected to multiple different ports, jacks, or other hardware connectors of the ECU under test 250 using wires. For example, while testing the ECU 250 and listening on the communications infrastructure 150, the test unit 210 can read outputs generated by the ECU 230, which may generate outputs on certain ports, such as pins indicating a reboot, based on its state or context. Such state or context may be automatically used by the test unit 210 to select messages to generate, identify a test failure or pass, etc., as described above. Identifying a test as failed, as described herein, may refer to detecting that at least one component of the system under test has malfunctioned or failed to operate as expected. Stated another way, identifying a test as failed may be based on identifying that a component in the system under test has deviated from expected performance, operation, or functionality. In some cases, the test unit 210 may identify the system under test as having passed the test even when some deviations are detected. For example, based on information from test procedure 271 or test 270, some deviations in some (perhaps insignificant) components may be recorded but others may be ignored, so that test unit 210 can still determine that the system under test passed the test.

[0053] Test procedures 271 may be initiated automatically by a remote or external system; for example, a remote computer may signal or command test unit 210 to begin testing, e.g., indicating which test procedures 271 to perform. A detail or example of test procedures 271 may be an "auto-discovery" procedure, which, when executed by test unit 210, automatically discovers components (e.g., ECUs, sensors, etc.) and services (e.g., software services) and automatically generates test procedures 271 including tests 270 covering some or all of the discovered components and services. For example, when some new version of ECU 230 or ECU 250 is compiled, deployed, and ready for testing, a remote computer may signal test unit 210 to begin test procedures 271. Test procedures 271 may include any information or data required by test unit 210 to execute the procedure; for example, test procedures 271 may include a list of tests 270 to execute, the order in which to execute tests 270, what to do if a test fails, etc.

[0054] One embodiment of the present invention is a testing method that includes automatically initiating a test procedure for a system under test, the procedure including transmitting at least one message over a network internal to the system under test. For example, upon being instructed to test a system under test 260, a testing unit 210 automatically selects a set of tests 270 and automatically initiates a test procedure for the system under test 260, the procedure including the selected tests 270.

[0055] The test procedure 271 (or one test procedure 271) may include information used by the test unit 210 to automate testing, as described above. For example, the test procedure 271 may include a list of tests 270 to run, the order in which to run the tests 270, and the test procedure may specify what to do when a component fails a particular test 270, when to start a test, conditions to stop or abort a test, etc. The test unit may automatically generate the test procedure 271. For example, once the test unit 210 automatically discovers or identifies a component, ECU, or service, as described above, it may generate the test procedure 271 by selecting tests 270 associated with the discovered component, ECU, or service and including the selected tests 270 in the newly automatically generated test procedure 271.

[0056] The test unit 210 can automatically start and stop a test procedure, for example, the test procedure 271 can start at a particular (e.g., preset) time or date, or the test unit 210 can automatically select the number of times the test procedure 271 is run. The test unit 210 can automatically start and stop tests based on configuration data, for example, the test unit 210 can automatically select tests 270 to include in the test procedure based on attributes of the ECU 230, for example, the type of ECU 230, an identification value that identifies the ECU 230, etc.

[0057] As referred to herein, the type of ECU may be, for example, an engine ECU, a body ECU, an infotainment ECU, a telematics ECU, a gateway ECU, an airbag ECU, an electronic power steering ECU, etc. The type of ECU or other component may be obtained, for example, using an identification (ID) value or parameter. For example, test unit 210 may obtain the ID from ECU 250 (or from ECU 230) and use this ID to search a database in storage 130 from which to easily retrieve a data sheet or other material describing the ECU or component.

[0058] The test unit 210 can automatically initiate a test procedure when it detects that the system under test 260 is in a particular state or when it detects that the state of the system under test 260 has changed. For example, when the test unit 210 is connected to multiple ports in the ECU 230 over the connection lines 290, the test unit 210 can detect or identify when the ECU 230 has stopped functioning, crashed, restarted, or malfunctioned, or when the ECU 230 has crashed, restarted, or completed a boot sequence. Thus, the test unit 210 can automatically initiate a test procedure 271 based on or in response to a state or event associated with the system under test 260, as well as based on other aspects, such as the time, date, or configuration data.

[0059] For example, certain communication behaviors are expected for certain functions of an automotive system that implements a standardized automotive protocol. Such behaviors are reflected in messages sent by various components within the system. Thus, by mapping various messages or message flows to states, procedural flows, or contexts, the test unit 210 can identify or determine the state or context of a system under test based on messages received from the system under test, as described herein.

[0060] For example, messages sent during startup time, or messages that are service announcements, or messages related to service scans or subscription requests, can all be monitored, recorded, and processed and used to determine the current state of the system under test, and, as mentioned above, can be used to identify deviations or failures that occur as a result of the test.

[0061] The test unit 210 can automatically cause the transmission of at least one message over a network internal to the system under test. For example, the test unit 210 can send a message over the connection line 280 into the communication infrastructure 253 internal to the system under test 260, thus causing the transmission of the message over the network internal to the system under test. Similarly, the test unit 210 can send a message over one of the connection lines 290 into the communication infrastructure 150 internal to the ECU 230.

[0062] In some embodiments, the test unit 210 can indirectly cause messages to be sent over the internal network. For example, the test unit 210 can send a first message to a first component in the system under test 260, causing that component to send a second, different message over the internal network 253. For example, the test unit 210 can send a message to one of the sensors 251, causing that sensor to send a message to one of the ECUs 250, e.g., to test how the ECU 250 reacts to the message from the sensor 251. In another example, the test unit 210 can send a first message to a first software component, e.g., a service or application executed by the controller 105 in the ECU 230 over the internal communication infrastructure 150, causing that software component to send a second, different message over the internal communication infrastructure 150 in the ECU 230.

[0063] In another example, the test unit 210 may interact with the system under test, for example by sending a signal to the ECU 230 on one of the connection lines 290 (e.g., by changing the voltage from "0" to "1"), causing a message to be sent on an internal network, and the test unit 210 may cause the ECU 230 to perform various actions, which may be, cause, or include sending a message on a network internal to the ECU 230, such as the communications infrastructure 150.

[0064] Test unit 210 may receive data communicated by components within the system under test. Generally, when connected to internal communication infrastructures 150 and 253 within computing device 100 and system under test 260, respectively, for example, over connections 280 and 290, test unit 210 may receive all messages communicated over these internal communication infrastructures. For example, test unit 210 may eavesdrop or eavesdrop on communication infrastructure 150 or communication infrastructure 253 using a sniffer mode, as known in the art. Test unit 210 may use connections 280 and 290 to transmit messages over internal communication infrastructures 280 and 290. For example, when test unit 210 is connected to network 253, it may receive messages exchanged between ECU 250 and sensor 251, and when test unit 210 is connected to communication infrastructure 150, it may receive messages exchanged between software modules, units, or components within computing device 100 or ECU 230.

[0065] Based on the received data, the test unit 210 can automatically identify deviations from the expected behavior of the system under test. For example, the controller 105 in the test unit 210 may expect a response message to be received within a predetermined time after a message it sends on the network 253 in the system under test 260, and if the response message is not received as expected, the controller 105 in the test unit 210 may identify that a deviation from the expected behavior has occurred.

[0066] Test unit 210 may automatically identify deviations from expected behavior of the system under test. For example, the expected behavior of ECU 230 may include sending a set of messages after or in response to sending a particular message on bus 150 by a service performed by controller 105 in ECU 230, and if such a set of messages is not sent on bus 150 after the particular message is sent on bus 150, test unit 210 may identify a deviation from expected behavior.

[0067] The test unit 210 may automatically identify deviations from a protocol. For example, a communication protocol may specify a sequence of messages that must be sent and / or received to conform to the protocol, and the test unit 210 may verify that a particular communication or other protocol is being observed or conformed to by inspecting messages transmitted on the internal network of the system under test. If the system under test identifies or determines that it is not observing or conforming to the protocol, the controller 105 within the test unit 210 may determine that a deviation from the protocol has occurred.

[0068] For example, the TCP / IP protocol specifies that upon receiving a message, the receiver must send an ACK message, and if, for example, the test unit 210 identifies a message sent from, for example, one of the ECUs 250 to one of the sensors 251, but the sensor 251 does not ACK the message for a predetermined period of time, the test unit 210 may determine a protocol violation. Other deviations from the TCP / IP protocol may be an incorrect sequence number, an incorrect source / destination address, an incorrect message header size, etc.

[0069] The testing unit 210 may automatically associate a deviation with a particular test. For example, the testing unit 210 may record a deviation such as that described above that occurred during the third test 270 in the testing sequence and may report the recorded information to the user.

[0070] In some embodiments, generating traffic on the internal network of the system under test includes selectively randomizing data in messages sent on the internal network. For example, the test unit 210 randomizes one or more of the size, destination, and content of the messages. To randomize the sent data, the test unit 210 can randomize any fields in the message headers as well as the actual content or payload within the messages. In some embodiments, the test unit 210 randomizes message attributes such as headers and content using fuzzing techniques as known in the art.

[0071] In some embodiments, test unit 210 automatically discovers or identifies at least one service provided by at least one component in the system under test based on data received from the system under test. For example, a particular service may advertise the service using a particular IP port number, e.g., by broadcasting the service. Thus, in one embodiment, test unit 210 discovers and identifies services provided by components in the system under test by sniffing the internal network as described above (or eavesdropping as known in the art). In some cases, once test unit 210 discovers, learns, or identifies a service as described above, it records (and / or reports) any information related to the service, such as the address of the component providing the service, the type of component providing the service (e.g., the type of ECU 250 or sensor 251).

[0072] In some embodiments, the test unit 210 automatically discovers and identifies components within the system under test, i.e., the test unit 210 automatically determines the type and function of components included in the system under test. For example, the test unit 210 identifies at least some of the ECUs 250 by receiving, inspecting, and processing broadcasted or other messages advertising services or components within the system under test 260, i.e., the test unit 210 determines that these ECUs 250 are included in the system under test 260. In addition to determining whether an ECU 250 is present and / or active within the system under test 260, in some embodiments, the test unit 210 automatically and specifically identifies the ECUs. For example, by sending various messages, e.g., according to a particular protocol, the test unit 210 determines that a first ECU 250 within the system under test 260 is designed to control the vehicle's brakes and that a second ECU 250 within the system under test 260 is designed to control the vehicle's headlights.

[0073] For example, upon receiving identification information from an ECU 250 (e.g., a serial number, model number, or any other ID), the test unit 210 can access related information, such as a data sheet from the manufacturer. The ECU's ID can be associated with a particular test 270. Similarly, a service can identify itself, for example, as defined by protocols known in the art.

[0074] In another example, the test unit 210 probes the system under test in an attempt to discover components or services. For example, a particular ECU or service may be designed or behave in such a way that upon receiving a particular message, it responds with another particular message (e.g., as defined by a particular protocol), and thus, by sending (e.g., broadcasting) a particular message, the test unit 210 can discover ECUs, services, or other components in the system under test.

[0075] Automatically discovering and identifying services and components by test unit 210 as described above provides various advantages, for example, to save time, test unit 210 can exclude tests 270 from test sequence 271 that are designed for or specific to particular services or components that are not included in the system under test. By automatically excluding tests 270 from test sequence 271, test unit 210 can avoid running certain tests, thus improving the overall time it takes to complete a test cycle that covers all functionality of the system under test.

[0076] In some embodiments, test unit 210 automatically modifies the test procedure based on services or components discovered in the system under test. For example, upon discovering or learning that an ECU 250 that controls a vehicle's engine is present in system under test 260, test unit 210 selects a particular test 270 designed to test the operation of the engine control ECU, e.g., test unit 210 includes the particular test 270 in test procedure 271. Thus, in certain embodiments, test unit 210 automatically and dynamically modifies the test procedure based on components automatically discovered and identified in the system under test. In some embodiments, test unit 210 automatically and dynamically selects one or more tests to include or exclude from test procedure 271 based, for example, on services or components automatically discovered as described above.

[0077] For example, if a particular ECU 250 that was expected to be included in the system under test 260 is not identified (e.g., does not respond to a query message and no messages are received from that particular ECU 250), the test unit 210 automatically excludes from the test sequence one or more tests 270 designed to test that particular (but missing) ECU 250. In another example, if the test unit 210 discovers an unexpected ECU 250, it may automatically add one or more tests 270 to the test sequence 271 so that the test sequence covers the discovered component.

[0078] In some embodiments, the test unit 210 automatically associates particular components or services within the system under test with deviations from expected behavior or protocols. For example, once the test unit 210 identifies components and services during a particular test 270, it may send a particular message to a particular ECU 250 or a particular service within an ECU 230; if the test unit 210 does not receive an expected response to the particular message, the test 210 may identify that the particular ECU 250 or service is associated with the deviation, i.e., failed a particular test 270. Thus, the test unit 210 can pinpoint a faulty ECU, service, or component and associate it with the failure of a particular test 270. For example, the test unit 210 may generate a report indicating that a particular sensor 251 failed the third test 270 in the test sequence 271.

[0079] In some embodiments, the test unit 210 sends specific messages over a bus or network in the system under test and determines the state of a component based on the response, or the absence of a response. For example, the protocols followed by the ECUs 250, sensors 251, and services described above typically specify the response that the recipient should send when receiving a specific message. In one embodiment, the test unit 210 determines the state of a component by sending such specific messages and determines that the component is functioning properly if the expected response is received, or that the component has failed (crashed, restarted) if no response or an incorrect response (according to the specific protocol) is received.

[0080] In some embodiments, when the testing unit 210 identifies that a particular component has failed a particular test 270, it records the particular component and the particular test 270, excludes the particular test 270 for the currently running test procedure 271, creates a modified test procedure 271 accordingly, and continues with the modified test procedure 271 or resumes testing the system under test using the modified test procedure 271. Thus, in contrast to some known techniques, systems, and methods that typically stop testing upon a failure, embodiments of the present invention are able to identify multiple failures by dynamically changing the particular test 270 used in the dynamically modified test procedure.

[0081] In some embodiments, the test unit 210 identifies or determines that a deviation has occurred based on the absence of a message, unexpected content in a message, the absence of a message modification, or a discrepancy with an expected sequence of events. For example, in some communication media, such as a CAN bus, certain portions of communication packets are expected to be modified by a receiver on the network. For example, by monitoring these particular bits during communication and detecting that these bits remain unchanged, the test unit 210 determines that a deviation from the protocol has occurred and / or that the receiving unit has malfunctioned, e.g., crashed.

[0082] Specifically, part of the CAN bus protocol includes an "ACK bit." When the system receives a transmitted packet, it acknowledges it by changing this bit to a "1." If the intended receiver crashes or otherwise malfunctions, the ACK bit remains unchanged, which can be identified by test unit 210 as a malfunction, deviation, failure, or breakdown. Stated another way, a "0" when and if a "1" is expected can be identified by test unit 210 as unexpected content in the message.

[0083] For example, the absence of a message identified by test unit 210 as a deviation or failure is the absence of an expected acknowledgement message (or ACK, as known in the art) (according to a particular protocol).

[0084] For example, certain communication protocols include queries to inquire (retrieve) metadata or other identifying information about a system. For example, a particular protocol known as Diagnostic Protocol over IP (DoIP) defines "identification queries" to which the recipient is expected to respond with a specific response message.

[0085] By issuing such requests while executing a particular test sequence, the test unit 210 can detect the exact point in the test sequence where the system under test failed. For example, during a test procedure 271 that includes executing multiple tests 270, following the completion of each test 270, the test unit 210 may send query messages to some of the ECUs 250 or some services in the ECU 230, thus ensuring that the ECUs 250 or services are still up and running and functioning properly. If a component does not respond as expected after a particular test 270, the test unit 210 may identify (and record and notify) that the component failed that particular test 270. Thus, embodiments of the present invention can associate specific components with specific tests, thereby pinpointing a failure of a specific component associated with a particular test. In other embodiments, the test unit 210 periodically and / or continuously pings, probes, or otherwise checks some or even all of the components and services in the system under test, for example, the test unit periodically and / or continuously sends query messages to components in the system under test, thus determining exactly when a component has stopped functioning.

[0086] In some embodiments, the test unit 210 receives a list of components included in the system under test, associates each component in the list with an automatically identified component in the system under test, e.g., based on a message from the component, and if any component in the list is not associated with a component in the system under test, the test unit 210 updates a log and / or alerts a user. Thus, the test unit 210 can automatically verify that all components expected to be in the system under test are actually there. For example, the manufacturer of the ECU 230 may provide a list of services provided by the ECU 230, and the test unit 210 sends messages using such list, which causes the service to respond with a specific message according to the relevant protocol. Thus, the test unit 210 can perform an automatic inventory check for the system under test.

[0087] In some embodiments, test unit 210 automatically generates messages to send on an internal bus or network based on previously automatically discovered services or components. In some embodiments, test unit 210 automatically generates messages to send on an internal bus or network based on previously automatically discovered component types and functions. For example, after automatically discovering services on ECU 230, test unit 210 sends messages to the discovered services to verify that they are functioning properly.

[0088] In some embodiments, test unit 210 sends messages to, for example, a particular ECU 250 or a particular service within ECU 230 after and based on the type and function of the particular ECU 250 or particular service, e.g., automatically identified as described above. For example, if test unit 210 determines that the discovered ECU 250 controls an air conditioning system, it sends a message to which the discovered ECU 250 is expected to respond in a particular known manner, e.g., acknowledging the message, performing an action, reporting a result, etc., according to, for example, a standardized protocol or data from the manufacturer.

[0089] In another example, when test unit 210 discovers and identifies a particular service contained in or running on ECU 230, it automatically adds tests 270 designed to test that particular service, e.g., requesting a particular operation or service provided by the automatically discovered service. Thus, test unit 210 is adapted to automatically discover components and services in the system under test, identify the type and function of the discovered elements, and adapt test procedures to test the discovered elements, providing a fully automated test system.

[0090] In some embodiments, the test unit 210 generates messages to be transmitted on a bus or network internal to the system under test based on a protocol followed by a component in the system under test, based on a service provided by a component in the system under test, or based on the function of a component in the system under test. For example, the test unit 210 generates messages based on the function of a particular ECU, based on a particular service provided by an ECU, or based on any component or element in the system under test. Thus, by generating service / ECU-specific messages, the test unit 210 can test a particular service / ECU.

[0091] In some embodiments, the test unit 210 identifies a procedure or a particular test 270 that resulted in a deviation from expected behavior or protocol. For example, some procedures occurring in a vehicle can be tracked based on a flow of messages, such as a first message (from the service requester) requesting service, a second message (from the service provider) approving the request, and a third message (from the service provider) reporting the results or conclusion of the operation or service. In one embodiment, the test unit 210, when provided with the associated protocol and any other relevant data and with messages sent over the internal network as described above, tracks the flows and identifies whether the flows are associated with a deviation, malfunction, or failure (e.g., the service provider crashed and stopped responding) or whether the flows are associated with expected behavior, such as whether the service provider provided the service successfully. Upon identifying a deviation, the test unit 210 can record the specific flow that exhibited the deviation, as well as other data, such as the specific test 270, the components involved, etc. Any data recorded by the test unit 210 may be provided to the user by the test unit 210, for example, presented on a screen, emailed, etc.

[0092] The term protocol referred to herein may refer to any protocol and / or communication standard used by the manufacturer of the system under test. For example, with respect to the automotive industry, the protocol may be SOME / IP, XCP, DoIP, or UDS.

[0093] Examples of expected behavior referred to in this specification may be those of in-vehicle components or systems operating according to protocols and communication standards used in the automotive industry, such as standards such as CAN or Ethernet, as well as protocols such as SOME / IP, XCP, DoIP, and UDS.

[0094] In some embodiments, the system under test (e.g., system under test 260) is designed and / or adapted to be installed in a vehicle, and the network within the system under test (e.g., communications infrastructure 253) is an in-vehicle network.

[0095] In some embodiments, test unit 210 identifies or determines a failure or unexpected behavior (deviation) by determining that a hardware component has stopped functioning properly. For example, a particular protocol may specify that a beacon message be transmitted by one of ECUs 250 at known intervals (e.g., every 15 milliseconds), and test unit 210 may set a time upon receipt of the beacon message and determine that a particular ECU 250 has crashed or stopped operating properly if the time expires before the next beacon message is received.

[0096] In some embodiments, test unit 210 identifies or determines a failure or unexpected behavior (deviation) by determining when a software component has stopped functioning properly. For example, test unit 210 may continuously and / or periodically probe or ping a service or other software component within ECU 230 or one of ECUs 250 and, if there is no response to the ping or probe message, determine that the service or software has crashed or is otherwise not functioning properly.

[0097] In some embodiments, the test unit 210 automatically selects tests 270 to include in the test sequence 271 based on the number and type of components in the system under test. For example, once the test unit 210 automatically identifies a particular group of ECUs 250 in the system under test 260, it automatically selects one or more tests 270 to use to test the system under test 260. For example, a particular ECU type may be associated with a particular test 270 in the database in the storage system 130, such that when that type and number of ECUs are identified in the system under test, test 270 selection is fully automated. Similarly, a particular test 270 may be associated with a particular software component; for example, a particular service running on an ECU 230 may be mapped to a particular test 270, thus enabling not only fully automated component discovery but also fully automated component testing. In some cases, the test unit 210 is provided with a list of components included in the system under test, and using the component-to-test mapping, the test unit 210 automatically defines and executes test procedures, i.e., selects tests 270 to be included in one or more test procedures and executes the test procedures.

[0098] In some embodiments, the test procedure 271 includes multiple tests 270, and the test unit 210 automatically determines whether the system under test (e.g., the ECU 230) passed at least one of the tests 270. For example, if a service in the ECU 230 stopped responding, the test unit 210 may determine that it failed that test. Any deviation as described above may cause the test unit 210 to determine that a test did not pass. The test unit 210 may take action if it determines that the system under test failed a test. For example, the action may include notifying or warning a user, restarting the test, rebooting the system under test and restarting the test, selecting a different test 270 for a subsequent test procedure, etc. For example, if one of the ECUs 250 in the system under test 260 fails a particular test 270 in the test sequence 271, the testing unit 210 may modify the test sequence and repeat the testing of the system under test 260 without testing the particular ECU 250 (which is already known to be defective). By excluding a test from the set of tests in the sequence in this manner, testing automation is improved; for example, by avoiding an ECU that is already known to be faulty, the testing unit 210 may move on to testing other components in the system under test 260. Stated another way, embodiments of the present invention can or may dynamically modify the test sequence 271 based on the results of a particular test; for example, dynamically modifying the test sequence may include excluding tests for failed or missing components, or dynamically modifying the test sequence 271 may include adding tests 270 to the test sequence 271 based on automatically identifying components in the system under test, as described herein.

[0099] In some embodiments, the test procedure includes a group of tests 270, and the testing unit 210 automatically identifies the state of the system under test after performing at least one of the tests. For example, the test procedure may include running five different tests 270, and the testing unit 210 may check and report the results (e.g., fail / pass) after each of the five tests 270. Thus, the user can be informed of exactly which test in a test group caused the failure.

[0100] In some embodiments, the test unit 210 may send messages over an internal network or bus, such as over infrastructure 150 or 253, according to a particular protocol, and / or procedure, and / or component, and / or structure of the system under test. For example, based on the type of one of the ECUs 250, the test unit 210 may select a particular protocol to use when communicating with the particular ECU 250. For example, as described above, using data sheets and other documentation from the manufacturer, the protocols and services supported and provided by a particular ECU type may be known and therefore may be readily used once the type or ID of the ECU or service is known, either if the type or ID is provided as configuration data or if the type or ID is automatically discovered as described herein.

[0101] In some embodiments, deviations from expected behavior of components of the system under test and / or deviations from protocols or communication standards used by the system under test and / or vulnerabilities in the system under test are identified based on network activity within the system under test. For example, when connected to an internal network or bus of the system under test, the test unit 210 identifies deviations or vulnerabilities based on the flow of messages between parties within the system under test; in this way, the test unit 210 identifies incorrect (or prematurely terminated) message flows. The incorrect or prematurely terminated flows or protocol violations can be associated by the test unit 210 with vulnerabilities or malfunctions.

[0102] An agent, as referred to herein and in the art, is a (typically small) software component that is installed on a system under test. The agent collects information within the system under test and transmits that information to a test unit. Using an agent requires effort, such as developing the agent and installing the agent on the system under test. Using an agent can produce inaccurate test results because the system under test (including the agent) is different from a later-manufactured system that does not include the agent.

[0103] As described above, the test unit 210 can monitor the system under test as a black box. For example, when connected to the system under test as described above, the test unit 210 does not need to install or require an agent on the system under test. Thus, embodiments of the present invention provide advantages over known systems and methods that require an agent.

[0104] As previously described, using detailed knowledge and understanding of automotive protocols, test unit 210 can derive the state of the system under test. For example, by tracking messages within a message flow, test unit 210 can identify deviations from the expected flow and log and report failures. For example, a protocol or service definition may indicate a specific response message that should be sent in response to a particular request message; if test unit 210 receives a defined request but does not receive a response defined by the protocol or service, test unit 210 may identify that a deviation has occurred and / or that a vulnerability has been detected and / or the system under test has failed the test.

[0105] As noted above, the methods described herein may be performed automatically by the test unit 210, i.e., without (or with minimal) intervention from a human tester or operator. For example, after connecting the test unit 210 to the system under test 260 (e.g., plugging it into a connector), the test unit 210 may automatically test the system under test 260 without further operator intervention. For example, the test unit 210 may automatically discover components within the system under test 260 and test the discovered components, and the test unit 210 may automatically select tests 270 to test the system under test. Such automation significantly reduces the time, effort, and resources required to scan a system under test for vulnerabilities. Accordingly, embodiments of the present invention reduce the cost of testing and scanning for vulnerabilities, require less expertise from operators or employees, and improve test coverage.

[0106] For example, an operator or employee need only know how to connect the test unit 210 to the system under test 260; the operator or employee need not know anything about the system under test 260. By automatically selecting tests 270 based on coverage, test results, or the state of the system under test 260, embodiments of the present invention may extend test coverage, e.g., repeat or continue testing until all components, flows, services, or other relevant aspects of the system under test have been tested. An initial configuration may be received by the test unit 210. For example, a list of components or services expected to be included in the system under test, metadata associated with the components, specific test procedures 271 to be performed, etc., may be provided by a user using, e.g., a keyboard, mouse, etc. By reducing the time, labor resources, and knowledge required for testing, embodiments of the present invention enable testing at an earlier stage of development, thus providing yet another advantage.

[0107] Additionally, by automatically discovering specific hardware, services, or applications contained in or running on the system under test, the test unit 210 can specifically target components discovered during a vulnerability scan and avoid wasting time scanning for hardware, services, or other applications that are not contained in or do not exist within the system under test.

[0108] The automatic discovery performed by test unit 210 as described above dramatically improves the product testing cycle because test unit 210 can automatically and repeatedly test (and retest) the product (e.g., while an in-house version is released for testing), discover components and services, and test the discovered elements. Thus, by testing the system under test as described above, defects or vulnerabilities can be discovered early in the development or production stages. For example, as soon as a software version containing a new service is installed on ECU 230, test unit 210 can start test sequence 271, automatically discover and identify the new service, automatically test the newly added service, and select several tests 270 to be added to test sequence 271, e.g., so that the newly added service is tested. Similarly, a new ECU 250 added to system under test 260 can be automatically discovered, identified, and tested as described above.

[0109] 4, a flowchart of a method according to an exemplary embodiment of the present invention is shown. As indicated by block 410, the method or flow may include automatically initiating a test procedure for a system under test, which procedure includes transmitting at least one message over a network internal to the system under test. For example, the test unit 210 may automatically initiate a test upon connection to the system under test 260 or ECU 230, the automation may include automatic discovery of components and services, the automation may include automatic selection of tests 270 to run on the system under test, and the automation may include decision-making, such as stopping a test upon failure, removing a test from a test suite and restarting a test, adding a test to a test sequence, and running a new test with the added test. For example, if a particular ECU 250 fails a first test procedure 271, the testing unit 210 may modify the test procedure so that it does not test the failing ECU 250, thereby allowing the modified test procedure 271 to be used to test other components in the system under test 260 during proper operation of the particular ECU 250. In other words, the testing unit 210 may automatically avoid testing components or services that are known to have failed a previously executed test 270 or test procedure 271. The testing unit 210 may automatically modify the tests 270 and / or test procedure 271 such that it selectively skips or avoids testing components or services that are known to have failed a test, thereby allowing the remainder of the system under test to be tested while the skipped or avoided services and components operate normally, unimpeded.

[0110] As indicated by block 415, the method may include receiving data communicated by components within the system under test. For example, when test unit 210 is connected to the system under test over a communication line, bus, or network (e.g., over connection line 280), it receives any messages or data communicated over communication infrastructure 253 as previously described.

[0111] As indicated by block 420, a method performed by an embodiment of the present invention may include identifying, based on the received data, a deviation from the expected behavior of at least one component of the system under test and at least one of the protocols or communication standards used by the system under test. For example, knowing that a component is expected to respond to a particular message or event with a particular message or action, the test unit 210 identifies the deviation. For example, according to SOME / IP or DoIP used by the ECU, the ECU is expected to respond to a query with a particular response, but if the ECU does not respond as expected, the test unit 210 identifies that a deviation from the protocol has occurred. A deviation may be a deviation from expected behavior or function. For example, perhaps upon starting the engine, the test unit 210 may expect the engine ECU to send a query to the oil pressure sensor (or ECU); if such a message is not seen, received, or detected as described, the test unit 210 identifies that a deviation from expected behavior or function has occurred.

[0112] As indicated by block 425, methods performed by embodiments of the present invention may include automatically associating a particular test with a deviation. For example, if a particular ECU 250 fails a particular test 270 in a test procedure as described above, the test unit 210 may record the particular test 270 associated with the failed ECU 250 and thus notify a user that the particular ECU 250 failed the particular test.

[0113] Other embodiments relate to the following: (1) In the test method, automatically initiating a test procedure for the system under test, the procedure including causing at least one message to be transmitted over a network internal to the system under test; receiving data communicated by components within the system under test; Based on the data received, the expected behavior of at least one component of the system under test; and The protocol or communication standard used by the system under test and identifying deviations from at least one of Automatically associating a specific test with a deviation; Includes. (2) The method of (1), comprising generating traffic on the internal network by selectively randomizing data in messages sent on the internal network. (3) The method of (1) or (2), including automatically identifying at least one service provided by at least one component in the system under test based on the received data. (4) Any one of the methods (1) to (3), including automatically identifying and specifying the types and functions of components included in the system under test. (5) Any one of the methods (1) through (4), including automatically modifying the test procedure based on at least one of the services and components discovered in the system under test. (6) Any one of the methods (1) to (5), including automatically associating a particular component or service with the deviation. (7) sending messages within the system under test and responses and absence of responses; and determining the state of the component based on at least one of the following: (8) Deviations are Absence of messages, Unexpected content in the message, Absence of message changes, A mismatch with the expected sequence of events Any one of methods (1) to (7), which is at least one of the above. (9) receiving a list of components included in a system under test; Associating each of the components in the list with an automatically identified component on the network based on a message from the component; updating the log and / or alerting the user if any of the components in the list are not associated with a component on the network; Any one of methods (1) to (8) including: (10) At least one message was automatically identified services provided by components of the system under test; The types and functions of components included in the system under test Any one of the methods (1) to (9) is automatically generated based on at least one of the following: (11) At least one message includes a protocol to which the component in the system under test conforms, a service provided by the component in the system under test, and a function of the component in the system under test. The method according to any one of (1) to (10), wherein the image is generated based on at least one of the following: (12) Any one of the methods (1) to (11) including identifying procedures or tests that result in deviations from expected behavior. (13) Any one of the methods (1) to (12), wherein the protocol and communication standard is used by the manufacturer of the system under test or by the vehicle industry. (14) Any one of the methods (1) to (13), wherein the system under test is designed / adapted to be mounted in a vehicle, and the network inside the system under test is an in-vehicle network. (15) Deviations may include malfunctioning hardware components and malfunctioning software components. Any one of methods (1) to (14), including one of the following: (16) The test groups to be included in the test procedure are identified by the components of the system under test and the services provided by them (either automatically discovered or provided as a list). Any one of methods (1) to (15), including selecting based on the number and type of (17) The procedure includes multiple tests, and the method is automatically determining a state of the system under test after performing at least one of the tests; automatically determining whether the system under test passed the at least one test; Any one of methods (1) to (16) including: (18) The transmission of at least one message is based on the protocols, procedures, components, and / or structure of the system under test. Any one of methods (1) to (17) according to at least one of the following: (19) Deviation from expected behavior may be achieved by any one of (1) to (18), including crashing or restarting the component. (20) In the test unit, automatically initiating a test procedure for the system under test, the procedure including causing at least one message to be transmitted over a network internal to the system under test; receiving data communicated by a component of the system under test; Based on the received data, the expected behavior of at least one component of the system under test; and The protocol or communication standard used by the system under test and identifying deviations from at least one of and automatically associating a particular test with the deviation.

[0114] In the detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, modules, units, and / or circuits have not been described in detail so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, discussion of identical or similar features or elements may not be repeated.

[0115] Although embodiments of the invention are not limited in this respect, for example, discussions using terms such as “processing,” “operating,” “calculating,” “identifying,” “establishing,” “analyzing,” and “checking” may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in the computer's registers and / or memory and / or into other data that is similarly represented as physical quantities in the computer's registers and / or memory or other information non-transitory storage medium that may store instructions for performing operations and / or processes. Although embodiments of the invention are not limited in this respect, as used herein, the term “plurality” may include, for example, “multiple” or “two or more.” The term “plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, etc. As used herein, the term group may include one or more items.

[0116] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or chronological order in time. In addition, some of the described method elements may occur or be performed simultaneously, contemporaneously, or concurrently. Some of the described method elements may be skipped or repeated during the sequence of operations of the method.

[0117] In the description and claims of this application, the verbs "comprise," "include," and "have," as well as their conjugations, are used to indicate that one or more objects of the verb do not necessarily completely exhaustively comprise components, elements, or portions of one or more subjects of the verb. Unless specifically stated otherwise, adjectives such as "substantially" and "about," modifying a property expressing a condition or relationship of one or more features of an embodiment of the present disclosure, should be understood to mean that the condition or property is defined within an acceptable tolerance for the operation of the described embodiment. Additionally, the word "or" is considered to be an inclusive, rather than exclusive, "or," indicating at least one or any combination of the items to which it connects.

[0118] The descriptions of embodiments of the present invention in this application are provided by way of example and are not intended to limit the scope of the present invention. The described embodiments include different features, not all of which are required in all embodiments. Some embodiments utilize only some of the features or possible combinations of the features. Variations of the described embodiments of the present invention, and embodiments including different combinations of the features set forth in the described embodiments, will occur to those skilled in the art. The scope of the present invention is limited only by the claims.

[0119] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art, and it is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

[0120] Various embodiments are presented, each of which may, of course, include features from the other presented embodiments, and embodiments not specifically described may include various features described herein. [Explanation of symbols]

[0121] 100 computing devices 105 Controller 115 Operating Systems 120 memory 125 executable code 130 Storage Systems 135 I / O Components 140 Input / Output Devices 150 Communications Infrastructure 210 Test Units 222 RAM 223 Flash Memory 230 ECU 240 Network 250 ECU 251 Sensors 253 Internal Communications Infrastructure 260 System Under Test 270 exams 271 Test Procedures 280 connection lines

Claims

1. In the test method, automatically initiating a test procedure for a system under test, said procedure including causing at least one message to be sent over a network internal to said system under test; receiving data communicated by components of the system under test; Based on the received data, an expected operation of at least one component of the system under test; and the protocol or communication standard used by the system under test; and identifying deviations from at least one of automatically associating a particular test with said deviation; A method comprising:

2. 10. The method of claim 1, comprising generating traffic on the internal network by selectively randomizing data in messages sent on the internal network.

3. The method of claim 1 , further comprising automatically identifying at least one service provided by at least one component within the system under test based on the received data.

4. 10. The method of claim 1, comprising automatically identifying and specifying the types and functions of components included in the system under test.

5. The test procedure is performed on the services and components discovered in the system under test. The method of claim 1 , further comprising automatically modifying the image based on at least one of:

6. The method of claim 1 , comprising automatically associating a particular component or service with the deviation.

7. 10. The method of claim 1, comprising: sending a message within the system under test; and determining a state of a component based on one of a response and an absence of a response.

8. The deviation is Absence of messages, Unexpected content in the message, Absence of message changes, A mismatch with the expected sequence of events The method of claim 1 , wherein the at least one of

9. receiving a list of components included in the system under test; Associating each of the components in the list with a component that is automatically identified on the network based on a message from the component; updating a log and / or alerting a user if any of the components in the list are not associated with a component on the network; The method of claim 1 , comprising:

10. The at least one message is automatically identified. Services provided by components within the system under test; and The types and functions of components included in the system under test The method of claim 1 , wherein the information is automatically generated based on at least one of:

11. 2. The method of claim 1, wherein the at least one message is generated based on at least one of a protocol conforming to a component in the system under test, a service provided by a component in the system under test, and a function of a component in the system under test.

12. The method of claim 1 , comprising identifying the procedure or test that caused the deviation from expected behavior.

13. The method of claim 1 , wherein the protocols and communication standards are used by a manufacturer of the system under test or by the vehicle industry.

14. The method of claim 1 , wherein the system under test is designed / adapted to be installed in a vehicle, and the network internal to the system under test is an in-vehicle network.

15. In the test unit, automatically initiating a test procedure for a system under test, said procedure causing at least one message to be sent over a network internal to said system under test; receiving data communicated by components of the system under test; Based on the received data, an expected operation of at least one component of the system under test; and the protocol or communication standard used by the system under test; and identifying deviations from at least one of automatically associating a particular test with said deviation; A test unit including:

Images