Fuzz testing software systems

Abstract

The present invention relates to fuzz testing of a software system. The method for fuzz testing the software system includes: determining a target number of times to test each callable unit; initializing a plurality of sorted queues; obtaining a seed from the highest sorted non-empty queue; performing a mutation process on the obtained seed to generate a test seed; providing the test seed as input to a software system for processing by the software system; evaluating the software system's processing of the test seed to generate a test result; and adding the test seed to the queue or discarding the test seed.

Description

Technical Field

[0001] The present invention relates to a method for fuzz testing a software system, and a system and a computer program for implementing such a method. Background Art

[0002] Software system testing is a vital component of software development and deployment. The large amount of code / instructions that comprise a software system means that "faults" (such as vulnerabilities, bugs, security weaknesses, or other issues) can be introduced (often accidentally) when writing code for the software system. If the software system is not tested, such faults can remain in the software system after deployment and can cause problems during the software system's execution. Such problems may be relatively harmless or inconvenient; others may cause unexpected / unintended behavior of the software system, including crashes; and still others may be more catastrophic and even result in loss of life (for example, if the software system is controlling or interacting with physical systems involving humans / animals). Some faults may provide attack vectors through which an attacker can execute one or more exploits, which can then result in problems such as loss of functionality, providing the attacker with unauthorized access to functionality / data, and so on, all with attendant costs and impacts.

[0003] Herein, a "software system" may be considered to be an entire / complete software (code / instruction) system; however, a "software system" may be a subsystem or component of a larger software system. Typically, a software system includes a plurality of "callable units" and is arranged to receive input for processing by the software system. Each "callable unit" may be, for example, a respective one of the following: a routine; a subroutine; a function; a procedure; a process; a class method; an interface; a component; or a subsystem of a larger system; and so on. References herein to a particular type of callable unit (e.g., reference to a "function" or a "component") should be considered to include references to other types of callable units.

[0004] The following discussion will focus on "software systems" being software systems used to (at least partially) control vehicle operation, such as software systems used to control the operation of an automated vehicle. However, it should be understood that the techniques and issues discussed herein are more broadly applicable to other types of software systems, and the description herein should not be considered limited to software systems used to (at least partially) control vehicle operation.

[0005] The automotive industry faces numerous safety challenges. Protecting drivers goes beyond equipping vehicles with seatbelts and airbags and extends to implementing appropriate safety measures to protect vehicles from malicious cyberattacks. Rapid advances in technology and connectivity have transformed the vehicle landscape. Modern cars are more than just mechanical devices controlled and driven by humans. They are connected autonomous vehicles (CAVs), combining infrastructure and computer processing with advanced wireless communications to make decisions and provide a safer, more engaging experience for drivers and passengers.

[0006] As the competition among original equipment manufacturers (OEMs) in autonomous driving and driver assistance continues, the opportunities for attackers to take control of vehicles have increased [1]. Software integration and connectivity make vehicles smart devices. However, this opens a window for software flaws and vulnerabilities that attract malicious behavior. In fact, vehicles with human drivers and autonomous driving or driver assistance features pose the greatest risk due to the maximized attack surface compared to fully manual, disconnected vehicles or fully automated vehicles. Internet exposure introduces a large number of vulnerabilities and facilitates the work of attackers. The threat posed by hackers in the automotive sector is not limited to breaches that only exploit personal data; they can amplify the risk by changing the vehicle software system. There are currently many documented attacks on vehicles launched by different vehicle manufacturers. Therefore, OEMs are working to strengthen their security measures to increase the resilience of vehicles to cyber attacks.

[0007] Since modern vehicle development relies on software, ensuring the development lifecycle is an important task to provide consumers with a better experience. Different standards such as Automotive Open System Architecture (AUTOSAR), J3061 [4] and ISO 26262 emphasize the importance of deploying security measures in all stages of vehicle software engineering (VSE) [6]. Since the need to develop secure vehicle software systems is higher than ever, the International Organization for Standardization (ISO) is collaborating with the Society of Automotive Engineers (SAE) to design the standard ISO / SAE 21434 specifically for secure development. This standard is designed to help OEMs address cybersecurity issues throughout the vehicle engineering lifecycle.

[0008] Before a vehicle is released, safety engineers need to verify the safety of the system to avoid catastrophic events. The lack of quality assurance and testing procedures in the automotive industry is one of the main factors that lead to the existence of vulnerabilities

[10] . Obviously, security testing is a key stage in VSE to identify vulnerabilities and system weaknesses. The automotive industry has adopted different security assurance methods, including static code analysis, dynamic code analysis, as well as vulnerability scanning, penetration testing, and fuzz testing

[11] . These security testing techniques can reduce the number of vulnerabilities in the system

[12] .

[0009] Regardless, safety testing of vehicle software systems is a complex task that presents multiple challenges to OEMs [6]. Vehicle software systems are complex systems with hundreds of millions of lines of code residing and running on dozens of electronic control units (ECUs)

[13] . These ECUs can operate based on inputs from radar, lidar, cameras, ultrasonic sensors, temperature sensors, tire pressure sensors, and many other sensors. As the vehicle operates in an evolving environment, the inputs to the ECUs can change dramatically. Therefore, it is difficult or impossible to predict all possible input combinations to the ECUs.

[0010] Some researchers

[10] ,

[15] ,

[16] consider fuzz testing to be one of the most suitable tools for discovering vulnerabilities in vehicle software systems. However, only a few companies have introduced fuzz testing tools designed explicitly for the automotive industry

[10] ,

[15] ,

[16] . Research work in this area is limited to evaluating and studying the applicability of black-box fuzz testing for CAVs

[19] ,

[20] . However, adopting this testing approach for safety-critical systems is not a reliable solution. Black-box random fuzz testing cannot provide a complete picture of which components are tested. For this reason, the automotive industry needs a software safety testing solution that can facilitate the testing process, simulate the vehicle environment, and locate vulnerabilities.

[0011] Safety testing is a powerful mechanism for detecting and identifying system vulnerabilities. In critical systems like vehicle software, software testing can prevent life-threatening accidents. However, many challenges make safety testing a complex undertaking in the automotive industry. Some of these challenges are outlined below.

[0012] System complexity and scale

[0013] Vehicle software systems include heterogeneous functions such as safety-related functions, infrastructure software, and multimedia systems [6],

[21] . The large number of operations that CAVs must perform increases the number of source code lines (SLOC) and the required hardware devices. Vehicle software systems are considered to be one of the largest existing systems

[23] . Safety engineers need to ensure the stable operation of the system, but this task becomes very time-consuming due to the large scale of the system. What makes the work of safety engineers more challenging is the complexity of the system. The heterogeneous functions of vehicle software systems use various advanced methods and technologies such as sensors, ECUs, network connectivity, artificial intelligence, data analysis, etc. All these components make the system a complex system and are expected to operate seamlessly and correctly. It is recognized that the design and development of complex code is challenging and leaves a lot of room for vulnerabilities and security issues

[24] -

[27] . Safety engineers must manage the complexity and size of the code to verify safety and ensure that the system does not reach a dangerous state throughout its operational life cycle.

[0014] Outsourcing

[0015] The development of heterogeneous functions embedded in vehicle software systems requires a wide variety of expertise and skills. Consequently, OEMs tend to outsource a large number of vehicle functions

[28] . While this may improve product quality, outsourcing complicates the work of security engineers. Software developed by third parties may introduce new threats and vulnerabilities to the system

[29] . This is made more difficult by layered and sometimes complex supply chains. Security engineers must work with applications and demonstrate their security and reliability without knowing the underlying development details or complete provenance of the applications. Furthermore, security testing and system failure rates should be applied to the entire system. Because many functions in a vehicle software system are interdependent, this process may be delayed until all components are fully integrated, significantly reducing the available testing and analysis time.

[0016] Input and output fluctuations

[0017] CAVs make rational decisions based on the surrounding environment to transport passengers safely to a specific destination. They may use one or more devices such as sensors, radars, lidars, and cameras to collect the required information to understand road conditions, weather conditions, and surrounding traffic

[30] . Evaluating all possible external environmental data sets is a difficult problem. Therefore, testing and validating the behavior of vehicle software systems is a challenging task. In addition to external data, ECUs also exchange internal data to trigger specific events. For example, the powertrain control module (PCM) controls the fuel consumption required to propel the vehicle. The PCM relies on different inputs to determine the correct mixture ratio, including engine temperature, air temperature, and throttle position. In modern vehicles, the PCM also receives internal information from the adaptive cruise control (ACC) ECU to control speed. Safety engineers must verify that the catastrophic failure rate of the system is within an acceptable range, which requires hours of intensive testing that should cover a large number of possibilities

[31] .

[0018] The complexity of the test platform

[0019] Test conditions greatly affect the accuracy of the results. The safety assurance and verification of the system should be performed under the same conditions as in real-world scenarios. Given the structure and complex architecture of vehicle software systems, simulating real-world environments becomes an expensive and time-consuming task

[30] . Vehicles operate in a wide range of different scenarios, including a variety of roads, speeds, visibility, density, communication modes, and drivers. Simulating one scenario may not be enough to ensure the safety and reliability of the system. Many industrial solutions provide OEM software-in-the-loop (SiL) and hardware-in-the-loop (HIL) test simulators that simulate real-world environments to evaluate vehicle software systems. However, several limitations prevent simulators from becoming a complete solution that can replace real-world testing of autonomous vehicles. Simulators are prone to errors and may not fully simulate real-world scenarios

[37] ,

[38] .

[0020] It should be understood that the above challenges and possible other challenges are also equally or similarly applicable to software systems for other purposes (ie, not only to software systems for vehicles).

[0021] Security protection is a closely related discipline in the automotive industry. Any security vulnerability in the vehicle software system may have a huge impact on the safety of the vehicle, making cybersecurity an indispensable task in VSE. During the security verification and validation phase, security engineers must ensure that the development and design of vehicle systems comply with the cybersecurity requirements of vehicle standards such as AUTOSAR, ISO 26262 and the upcoming ISO / SAE 21434 standard. This includes planning, reporting and, most importantly, a series of security tests to verify the protection mechanisms of the vehicle software system. Since vehicle systems incorporate various advanced technologies, including different communication methods and hardware devices, a variety of security testing techniques are needed to ensure the security of the system throughout its life cycle. Some testing techniques are automatically incorporated into the development process to identify potential weaknesses immediately, while other techniques require manual intervention and are run after the development phase

[11] . Some of the most common security assurance methods used in the automotive industry are: fuzz testing, penetration testing, static code analysis and vulnerability scanning. These are discussed in more detail below:

[0022] Static code analysis

[0023] Recommended by ISO 26262, static code analysis is a white-box testing method that dynamically and automatically analyzes the source code of a vehicle system to identify programming errors that make the system vulnerable to vulnerabilities

[39] . Imparato et al.

[40] examined the potential of existing static analysis tools to identify vulnerabilities in automotive software components. Their study showed that BugFinder and Polyspace Code Prover only identified a small number of code sections that did not meet safety protection standards, despite the high performance of these tools in other systems. Quality Accelerated (QA) tools performed better in identifying software defects that did not meet the MISRA coding standards developed by the Motor Industry Software Reliability Association. Coyle

[45] emphasized the importance of identifying race conditions in multi-threaded components of automotive software components - the authors proposed a static race condition code analyzer and demonstrated its potential in detecting serious defects that cause safety-critical systems to enter dangerous states.

[0024] Static code analysis tools can be run quickly during the development phase to identify various code defects that weaken the system. They are generally considered worthwhile, especially for MISRA compliance. However, these scanners have limitations. They have a high rate of false positives, which can waste security testers' time

[46] . Static code analyzers cannot find vulnerabilities in source code whose causes are not well understood and modeled (such as unchecked inputs and bounds), so other tools are needed.

[0025] Dynamic program analysis

[0026] Dynamic program analysis examines and monitors program execution to discover program reactions and identify incorrect behaviors. It covers all typical forms of software testing, including unit, component, integration, and system testing. From a safety perspective, it is used to find dangerous situations such as memory errors, concurrency errors, and crashes. Selick et al.

[47] promote program analysis techniques to identify security and privacy issues in Internet of Things (IoT) systems such as automotive systems. In their study, the researchers demonstrated the ability of dynamic program analysis to find vulnerabilities that other techniques (such as static code analysis) cannot identify. Kosher

[48] emphasized the severity of vulnerabilities in automotive systems and stressed the applicability of dynamic program analysis to quickly and easily identify automotive vulnerabilities. The researchers proposed a dynamic analysis tool that can simulate the inputs and outputs of embedded ECUs in near real time. Capodi et al.

[49] proposed a dynamic program analysis tool for automotive system safety testing that monitors and analyzes CAN message routing and filtering to identify unstable behaviors. Their case study of a gateway ECU demonstrated the effectiveness of the tool in minimizing workload and identifying abnormal reactions.

[0027] While dynamic program analysis can expose vulnerabilities that static code analysis cannot, it only covers known software issues. Dynamic program analysis operates against predefined scenarios, thus limiting the scope of testing. Furthermore, this approach to security testing assurance may not exercise all system components, limiting the vulnerability verification process to only certain code areas.

[0028] Vulnerability scanning

[0029] Vulnerability scanning verifies the resilience of vehicle software systems to known vulnerabilities and security flaws. In other words, such a security assurance method can detect development errors that cannot be fully traced but have associated attacks. This testing technique requires prior knowledge of attacks and security issues in the automotive industry. In 2015, leading industry pioneers collaborated to establish the Automotive Information Sharing and Analysis Center (AUTO-ISAC) to collect and analyze emerging cybersecurity risks in the automotive industry on a global scale. AUTO-ISAC provides OEMs with information on vulnerabilities identified by more than 30 automakers, enabling faster vulnerability detection and shared responsibility. In addition to improving the industrial strength of vulnerability scanning, researchers are also facilitating this process by integrating existing attacks. Ling et al.

[51] established a database of discovered vulnerabilities for access during the security verification and validation phase. Similarly, Summer et al.

[52] verified and classified automotive security attacks to enrich the security testing phase of VSE.

[0030] Vulnerability scanning is undoubtedly crucial for avoiding recurring attacks (including those discovered during penetration testing) and can be applied early in the development cycle. However, such security testing tools do not comprehensively assess a system. Systems developed by various parties possess unique weaknesses that vulnerability scanning cannot identify. Therefore, scans must be continuously customized for each specific system, requiring additional testing tools.

[0031] Penetration testing

[0032] To verify the resilience of vehicle software systems against malicious behavior, penetration testing can be performed. Penetration testing is the most studied testing technique in the automotive industry

[39] . Kosher et al.

[53] experimented with the security of vehicles by conducting a variety of physical and remote attacks. By simulating replay attacks, researchers were able to bypass basic network security protections within the vehicle. Xie et al.

[54] used penetration testing to evaluate the security of vehicle Bluetooth interfaces.

[0033] Other researchers have used penetration testing to assess the security of in-vehicle communications. Corbett et al.

[55] introduced a testing framework that attempts to bypass in-vehicle network intrusion detection systems (NIDS). Taylor et al.

[56] designed an anomaly detection framework for CAN buses. The researchers studied previous successful attacks to identify common characteristics and simulated a range of new attacks. Huang et al.

[57] verified CAN defense mechanisms by proposing a tool that automatically injects attack packets into the CAN bus.

[0034] While researchers have discovered several security vulnerabilities in vehicle systems through penetration testing, this testing method is highly effective in verifying vehicle cybersecurity. When performed well, penetration testing produces the most significant and meaningful results, but it is also the most time-consuming, least complete, and requires significant and rare expertise. Automation of known attacks has always been a crucial aspect of a functional penetration testing strategy in VSE. Combining all these techniques provides excellent coverage of well-known issues and attacks, as well as the most likely and significant ones. However, ensuring the resilience of vehicle software systems through penetration testing alone is insufficient.

[0035] Fuzz testing

[0036] Fuzz testing is a robust testing technique that verifies system behavior based on arbitrary inputs to identify unexpected behavior that an attacker can use to launch an attack

[58] . Three different testing approaches can be used: white-box, black-box, and grey-box fuzz testing.

[0037] Researchers in the automotive industry focus on black-box fuzz testing and avoid white-box fuzz testing. While white-box testing can provide a comprehensive assessment of a system, deploying such a mechanism in the automotive industry is time-consuming and labor-intensive given the complexity and scale of the systems. Furthermore, since many components of a vehicle's software system are outsourced, applying white-box testing to all of them is impractical.

[0038] Oka et al.

[19] considered black-box fuzz testing as one of the powerful tools for discovering vulnerabilities in vehicle software systems. They demonstrated its effectiveness by performing fuzz testing on engine ECUs and gateway ECUs. By monitoring the engine ECU’s response to fuzzy and random messages, the researchers successfully identified corrupted pulse width modulation (PWM) frequencies.

[0039] In another research work, Oka et al.

[59] highlighted the challenges of validating and testing complex and extensive systems, such as vehicle software systems. Starting testing after the system is completed can cause delays in vehicle production. Oka et al. found that fuzz testing allows testing to begin earlier in the development process. Random inputs can replace the inputs required to validate the developed functionality.

[0040] Similarly, Fowler et al.

[20] ,

[60] used an arbitrary controller area network (CAN) fuzz tester to identify security issues in ECUs. They performed black-box fuzz testing on the display ECU of a laboratory vehicle and demonstrated the benefits of fuzzing automotive inputs to identify bugs and weaknesses in the vehicle's software system.

[0041] While black-box fuzz testing can address system complexity, outsourcing, and input and output fluctuation challenges, blind testing of safety-critical systems carries risks. Black-box testing cannot guarantee good coverage or a thorough evaluation of the system. Furthermore, any test case may fail initial input validation requirements, preventing testing from extending to the core of the system. Adopting this testing approach in the automotive industry does not guarantee a risk-free lifespan.

[0042] Other gray-box fuzz testing techniques

[0043] Recently, gray-box fuzz testing has become a popular security testing tool

[61] . The most famous gray-box fuzz testing technique is American Fuzzy Lop (AFL). AFL collects coverage information to identify valuable test cases that extend code coverage. Various strategies have been introduced to enhance the coverage and performance of AFL

[63] –

[65] .

[0044] Existing grey-box fuzzing techniques are particularly ill-suited for systems like CAVs and their associated system complexity and scale challenges. They consume hours of testing time and are entirely focused on expanding code coverage.

[0045] Zhang et al.

[66] attempted to rank the seeds generated by AFL, but their test case prioritization did not guide testing in a specific direction. Bomei et al.

[65] introduced directed gray-box fuzzing (DGF), which focuses on testing a user-specified target. This goal is achieved by eliminating test cases that are far away from the target. They calculated the minimum distance between system nodes to identify close seeds. The minimum distance forms an important limitation because it eliminates critical paths in the system that may contain errors. DGF relies on prior knowledge of vulnerability-prone areas, which can be guided by threat and risk assessments, but not complete. In addition, when testing a newly developed system, the entire system must be examined rather than just specific functions. Summary of the Invention

[0046] Embodiments of the present invention aim to address the aforementioned shortcomings in software testing and security assurance. This goal is achieved through a gray-box fuzz testing framework that optimizes vulnerability exposure while addressing security testing challenges, such as those faced by the automotive industry. Gray-box fuzz testing is a robust security mechanism that accumulates information about a system without increasing testing complexity, enabling fast and efficient security testing. Embodiments of the present invention provide a vulnerability-focused fuzz testing framework that systematically prioritizes testing against vulnerable components of a software system (e.g., a vehicle software system). The framework utilizes security vulnerability metrics designed to identify vulnerable components in a software system and assign weights to ensure thorough testing of these components. Furthermore, in some embodiments, the mutation engine of some embodiments of the present invention can perform small data type mutations at high-level inputs to bypass input validation in some systems. Embodiments of the present invention can knowledgeably validate system components without increasing testing complexity, providing a security testing tool that efficiently and reliably manages a variety of testing challenges. Consequently, it expands vulnerability identification during the development phase and can enhance the resilience of software systems against unprecedented cyberattacks.

[0047] Gray-box fuzz testing allows for focused and efficient evaluation of software systems without analyzing every line of code. Unlike white-box testing, which employs intensive code analysis and constraint solving, gray-box testing does not incur high overhead. Furthermore, gray-box fuzz testing overcomes the randomness of black-box fuzz testing while rapidly generating a large number of test cases. Thus, the gray-box approach addresses three testing challenges: system complexity and scale by avoiding intensive code analysis, outsourcing by limiting system knowledge, and input and output volatility by creating a large number of inputs.

[0048] According to a first aspect of the present invention, a method for fuzz testing a software system is provided, wherein the software system includes a plurality of callable units and is arranged to receive input for processing by the software system, the method comprising: determining, for each callable unit in the plurality of callable units, a target number of times to test the callable unit based on one or more security vulnerability metrics; initializing a plurality of sorted queues, each queue being used to store one or more seeds, the initialization comprising storing the one or more initial seeds in corresponding queues in the sorted plurality of queues; executing a series of tests, wherein executing each test comprises: obtaining a seed from the highest sorted non-empty queue; performing a mutation process on the obtained seed to generate a test seed; providing the test seed as input to the software system for processing by the software system; and evaluating the software system's processing of the test seed to generate a test result; wherein each of the sorted plurality of queues has an associated seed addition. criteria, and wherein executing each test comprises: (a) adding a test seed to the highest-ranked queue of a plurality of ordered queues for which the test seed satisfies the seed addition criteria associated with the queue; or (b) discarding the test seed if the test seed does not satisfy the seed addition criteria associated with any queue of the plurality of ordered queues; wherein the seed addition criteria are configured such that: if processing of the first test seed by the software system involves execution of a callable unit of interest or an execution path close to the callable unit of interest and if processing of the second test seed by the software system does not involve execution of the callable unit of interest or an execution path close to the callable unit of interest, the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added, wherein a callable unit is a callable unit of interest if the current number of tests that result in execution of the callable unit is less than a target number of times the callable unit is to be tested.

[0049] In some embodiments of the first aspect, the seed addition criteria are configured such that: if the software system's processing of the first test seed involves an execution path close to the callable unit of interest but does not involve the execution of the callable unit of interest, and if the software system's processing of the second test seed involves the execution of the callable unit of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added. Alternatively, in some embodiments of the first aspect, the seed addition criteria are configured such that: if the software system's processing of the first test seed involves an execution path close to the callable unit of interest but does not involve the execution of the callable unit of interest, and if the software system's processing of the second test seed involves the execution of the callable unit of interest, then the queue to which the first test seed is added has a lower ranking than the queue to which the second test seed is added.

[0050] In some embodiments of the first aspect, the seed addition criteria are configured such that: if the software system's processing of a first test seed involves the execution of one or more first callable units of interest or is close to an execution path of one or more first callable units of interest and if the software system's processing of a second test seed involves the execution of one or more second callable units of interest or is close to an execution path of one or more second callable units of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added if: (a) the remaining number of times at least one of the one or more first callable units of interest is greater than the remaining number of times each of the one or more second callable units of interest is to be tested; or (b) the sum of the remaining number of times each of the one or more first callable units of interest is greater than the sum of the remaining number of times each of the one or more second callable units of interest is to be tested.

[0051] In some embodiments of the first aspect, a criterion for adding seeds to the first queue is that the processing of the test seed by the software system involves the execution of the callable unit of interest or is close to the execution path of the callable unit of interest. Additionally or alternatively, in some embodiments of the first aspect, a criterion for adding seeds to the second queue is that the processing of the test seed by the software system reaches a branch point in the software system that was not reached when executing a previous test. The first queue may have a higher ranking than the second queue. The sorted multiple queues may be a set including the first queue and the second queue.

[0052] In some embodiments of the first aspect, obtaining the seed from the highest-ranked non-empty queue includes removing the seed from the highest-ranked non-empty queue.

[0053] In some embodiments of the first aspect, the method includes determining a corresponding reuse quantity for a test seed, which indicates the number of future tests for which the seed can be used as the obtained seed. Determining the corresponding reuse quantity for the test seed may include: setting the reuse quantity to a first predetermined value if the software system's processing of the test seed involves the execution of a callable unit of interest; setting the reuse quantity to a second predetermined value if the software system's processing of the test seed does not involve the execution of the callable unit of interest but involves an execution path that is close to the callable unit of interest; and setting the reuse quantity to a third predetermined value if the software system's processing of the test seed does not involve the execution of the callable unit of interest or an execution path that is close to the callable unit of interest but reaches a branch point in the software system that was not reached when executing a previous test. In some such embodiments, either: (a) the first predetermined value is greater than the second predetermined value, and the second predetermined value is greater than the third predetermined value; or (b) the second predetermined value is greater than the first predetermined value, and the first predetermined value is greater than the third predetermined value. Additionally or alternatively, the method may include, for each stored seed, storing a corresponding reuse number, wherein obtaining the seed from the highest-ordered non-empty queue includes decrementing the reuse number corresponding to the seed and (a) retaining the seed in the highest-ordered non-empty queue if the reuse number corresponding to the seed is not zero, and (b) removing the seed from the highest-ordered non-empty queue if the reuse number corresponding to the seed is zero. Additionally or alternatively, adding a test seed to the highest-ordered queue of the ordered plurality of queues to test that the seed meets a seed addition criterion associated with the queue may include adding the test seed to the highest-ordered queue of the ordered plurality of queues to test that the seed meets the seed addition criterion associated with the queue a number of times equal to the reuse number, wherein obtaining the seed from the highest-ordered non-empty queue may then include removing the seed from the highest-ordered non-empty queue.

[0054] In some embodiments of the first aspect, performing a mutation process on the obtained seed to generate the test seed includes mutating the obtained seed to form the test seed.

[0055] In some embodiments of the first aspect, performing a mutation process on the obtained seed to generate a test seed includes: (a) if the obtained seed is an initial seed, setting the test seed to the obtained seed; and (b) mutating the obtained seed to additionally form a test seed.

[0056] In some embodiments of the first aspect, determining, for each callable unit in the plurality of callable units, a target number of times to test the callable unit may generate a higher target number when the one or more security vulnerability metrics indicate a higher level of security vulnerability for the callable unit.

[0057] In some embodiments of the first aspect, initializing the ordered plurality of queues includes storing each of the one or more initial seeds in a highest-ordered queue.

[0058] In some embodiments of the first aspect, the test sequence is executed until a termination condition is satisfied, wherein the termination condition comprises one or more of: (a) each queue in the ordered plurality of queues is empty; (b) a threshold number of tests has been executed; and (c) a threshold amount of time has been spent executing the test sequence.

[0059] In some embodiments of the first aspect, processing of a test seed by the software system is considered to involve an execution path approaching a first callable unit if the first callable unit is reachable from a furthest callable unit in a call graph of the software system, where the furthest callable unit is a callable unit in the execution path: no other callable unit in the execution path is further in the call graph from a root node in the call graph, and: (a) the number of callable units between the furthest callable unit and the first callable unit in the call graph is at most a predetermined threshold; or (b) the number of callable units between the furthest callable unit and the root node in the call graph is at least a predetermined threshold; or (c) an amount of code above the furthest callable unit in the call graph is at least a predetermined threshold; or (d) an amount of code below the furthest callable unit in the call graph is at most a predetermined threshold; or (e) an amount of code between the furthest callable unit and the first callable unit in the call graph is at most a predetermined threshold.

[0060] In some embodiments of the first aspect, the method includes providing output for the fuzz test based on results generated from the executed test.

[0061] In some embodiments of the first aspect, the software system is a software system of a vehicle.

[0062] In some embodiments of the first aspect, each callable unit is a respective one of: a routine; a subroutine; a function; a procedure; a process; a class method; an interface; a component; or a subsystem of a larger system.

[0063] In some embodiments of the first aspect, the one or more security vulnerability metrics include one or more of the following: (a) a metric representing the degree of security vulnerability and / or security criticality of a callable unit; (b) a metric representing the risk that a malicious message can be passed from one callable unit to another callable unit; (c) a metric based on the number and / or type of communication technologies used by the callable units; (d) a metric based on the code complexity of the callable units; (e) a metric based on the number of input and output parameters of the callable function that have varying values ​​and / or the degree to which the input and output parameters of the callable function can have varying values; and (f) a metric based on historical vulnerability data associated with the callable units.

[0064] According to a second aspect of the present invention, there is provided a test system for fuzz testing a software system, wherein the software system comprises a plurality of callable units and is arranged to receive input for processing by the software system, the test system comprises one or more processors, the one or more processors being arranged to: determine, for each of the plurality of callable units, a target number of times to test the callable unit based on one or more security vulnerability metrics; initialize a plurality of sorted queues, each queue being used to store one or more seeds, the initialization comprising storing the one or more initial seeds in a corresponding queue in the sorted plurality of queues; execute a series of tests, wherein executing each test comprises: obtaining a seed from the highest sorted non-empty queue; performing a mutation process on the obtained seed to generate a test seed; providing the test seed as an input to the software system for processing by the software system; and evaluating the software system's processing of the test seed to generate a test result; wherein the plurality of sorted queues Each queue has an associated seed addition criteria, and wherein executing each test includes: (a) adding the test seed to the highest-ranked queue among the sorted plurality of queues for which the test seed satisfies the seed addition criteria associated with the queue; or (b) discarding the test seed if the test seed does not satisfy the seed addition criteria associated with any queue among the sorted plurality of queues; wherein the seed addition criteria are configured such that: if the software system's processing of the first test seed involves execution of a callable unit of interest or an execution path close to the callable unit of interest and if the software system's processing of the second test seed does not involve execution of the callable unit of interest or an execution path close to the callable unit of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added, wherein a callable unit is a callable unit of interest if the current number of tests that result in execution of the callable unit is less than a target number of times the callable unit is to be tested.

[0065] In some embodiments of the second aspect, the seed addition criteria are configured such that: if the software system's processing of the first test seed involves an execution path close to the callable unit of interest but does not involve the execution of the callable unit of interest, and if the software system's processing of the second test seed involves the execution of the callable unit of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added. Alternatively, in some embodiments of the second aspect, the seed addition criteria are configured such that: if the software system's processing of the first test seed involves an execution path close to the callable unit of interest but does not involve the execution of the callable unit of interest, and if the software system's processing of the second test seed involves the execution of the callable unit of interest, then the queue to which the first test seed is added has a lower ranking than the queue to which the second test seed is added.

[0066] In some embodiments of the second aspect, the seed addition criteria are configured such that: if the software system's processing of a first test seed involves the execution of one or more first callable units of interest or is close to an execution path of one or more first callable units of interest and if the software system's processing of a second test seed involves the execution of one or more second callable units of interest or is close to an execution path of one or more second callable units of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added if: (a) the remaining number of times at least one of the one or more first callable units of interest is greater than the remaining number of times each of the one or more second callable units of interest is to be tested; or (b) the sum of the remaining number of times each of the one or more first callable units of interest is greater than the sum of the remaining number of times each of the one or more second callable units of interest is to be tested.

[0067] In some embodiments of the second aspect, a criterion for adding seeds to the first queue is that the processing of the test seed by the software system involves the execution of the callable unit of interest or is close to the execution path of the callable unit of interest. Additionally or alternatively, in some embodiments of the second aspect, a criterion for adding seeds to the second queue is that the processing of the test seed by the software system reaches a branch point in the software system that was not reached when executing a previous test. The first queue can have a higher ranking than the second queue. The sorted multiple queues can be a set including the first queue and the second queue.

[0068] In some embodiments of the second aspect, obtaining the seed from the highest-ranked non-empty queue includes removing the seed from the highest-ranked non-empty queue.

[0069] In some embodiments of the second aspect, the test system is arranged to determine a corresponding reuse quantity for the test seed, which indicates the number of future tests that the seed can be used for the obtained seed. Determining the corresponding reuse quantity for the test seed may include: setting the reuse quantity to a first predetermined value if the software system's processing of the test seed involves the execution of a callable unit of interest; setting the reuse quantity to a second predetermined value if the software system's processing of the test seed does not involve the execution of the callable unit of interest but involves an execution path that is close to the callable unit of interest; and setting the reuse quantity to a third predetermined value if the software system's processing of the test seed does not involve the execution of the callable unit of interest or an execution path that is close to the callable unit of interest but reaches a branch point in the software system that was not reached when executing a previous test. In some such embodiments, either: (a) the first predetermined value is greater than the second predetermined value, and the second predetermined value is greater than the third predetermined value; or (b) the second predetermined value is greater than the first predetermined value, and the first predetermined value is greater than the third predetermined value. Additionally or alternatively, the test system may be arranged to, for each stored seed, store a corresponding reuse number, wherein obtaining the seed from the highest-ordered non-empty queue comprises decrementing the reuse number corresponding to the seed and (a) retaining the seed in the highest-ordered non-empty queue if the reuse number corresponding to the seed is not zero, and (b) removing the seed from the highest-ordered non-empty queue if the reuse number corresponding to the seed is zero. Additionally or alternatively, adding a test seed to the highest-ordered queue of the ordered plurality of queues to test that the seed meets a seed addition criterion associated with the queue may comprise adding the test seed to the highest-ordered queue of the ordered plurality of queues to test that the seed meets a seed addition criterion associated with the queue a number of times equal to the reuse number, and obtaining the seed from the highest-ordered non-empty queue may then comprise removing the seed from the highest-ordered non-empty queue.

[0070] In some embodiments of the second aspect, performing a mutation process on the obtained seed to generate the test seed includes mutating the obtained seed to form the test seed.

[0071] In some embodiments of the second aspect, performing a mutation process on the obtained seed to generate a test seed includes: (a) if the obtained seed is an initial seed, setting the test seed to the obtained seed; and (b) mutating the obtained seed to additionally form a test seed.

[0072] In some embodiments of the second aspect, determining, for each callable unit in the plurality of callable units, a target number of times to test the callable unit may generate a higher target number when the one or more security vulnerability metrics indicate a higher level of security vulnerability for the callable unit.

[0073] In some embodiments of the second aspect, initializing the ordered plurality of queues includes storing each of the one or more initial seeds in a highest-ordered queue.

[0074] In some embodiments of the second aspect, the test system is arranged to execute the test sequence until a termination condition is satisfied, wherein the termination condition comprises one or more of: (a) each of the ordered plurality of queues is empty; (b) a threshold number of tests has been executed; and (c) a threshold amount of time has been spent executing the test sequence.

[0075] In some embodiments of the second aspect, processing of the test seed by the software system is considered to involve an execution path approaching the first callable unit if the first callable unit is reachable from a furthest callable unit in a call graph of the software system, where the furthest callable unit is a callable unit in the execution path: no other callable unit in the execution path is farther in the call graph from a root node in the call graph, and: (a) the number of callable units between the furthest callable unit and the first callable unit in the call graph is at most a predetermined threshold; or (b) the number of callable units between the furthest callable unit and the root node in the call graph is at least a predetermined threshold; or (c) the amount of code above the furthest callable unit in the call graph is at least a predetermined threshold; or (d) the amount of code below the furthest callable unit in the call graph is at most a predetermined threshold; or (e) the amount of code between the furthest callable unit and the first callable unit in the call graph is at most a predetermined threshold.

[0076] In some embodiments of the second aspect, the testing system is arranged to provide output for the fuzz testing based on results generated from the executed testing.

[0077] In some embodiments of the second aspect, the software system is a software system of a vehicle.

[0078] In some embodiments of the second aspect, each callable unit is a respective one of: a routine; a subroutine; a function; a procedure; a process; a class method; an interface; a component; or a subsystem of a larger system.

[0079] In some embodiments of the second aspect, the one or more security vulnerability metrics include one or more of the following: (a) a metric representing the degree of security vulnerability and / or security criticality of a callable unit; (b) a metric representing the risk that a malicious message can be passed from one callable unit to another callable unit; (c) a metric based on the number and / or type of communication technologies used by the callable units; (d) a metric based on the code complexity of the callable units; (e) a metric based on the number of input and output parameters of the callable function that have varying values ​​and / or the degree to which the input and output parameters of the callable function can have varying values; and (f) a metric based on historical vulnerability data associated with the callable units.

[0080] According to a third aspect of the present invention, a computer program is provided, which, when executed by one or more processors, causes the one or more processors to implement the method according to the first aspect or its embodiments. The computer program may be stored on a computer-readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0082] Figure 1 An example of a computer system is schematically shown;

[0083] Figure 2a Schematically illustrating framework steps according to some embodiments of the present invention;

[0084] Figure 2b Schematically shows a method for implementing Figure 2a The framework engine;

[0085] Figure 3 An example call graph is shown;

[0086] Figure 4 shows sample input for OpenPilot;

[0087] Figure 5 It is a graph that plots statement coverage curves to compare the three testing tools;

[0088] Figure 6 is a graph depicting crashes triggered by three testing tools;

[0089] Figure 7 is a chart comparing the number of crashes detected by the three testing tools and the number of times the weak components were tested;

[0090] Figure 8 is a Venn diagram showing the similarities between crashes reported by three testing tools;

[0091] Figure 9 is a flow chart illustrating a method according to some embodiments of the present invention;

[0092] Figure 10 schematically illustrates a plurality of queues in an ordered manner; and

[0093] Figure 11 An example use of an embodiment of the present invention is schematically illustrated. DETAILED DESCRIPTION

[0094] In the following description and in the drawings, certain embodiments of the present invention are described. However, it should be understood that the present invention is not limited to the described embodiments, and some embodiments may not include all of the features described below. However, it will be apparent that various modifications and variations may be made herein without departing from the broader spirit and scope of the present invention as set forth in the appended claims.

[0095] 1 – System Overview

[0096] Figure 1 Schematically, an example of a computer system 100 is shown. System 100 includes a computer 102. Computer 102 includes storage media 104, memory 106, processor 108, interface 110, user output interface 112, user input interface 114, and network interface 116, which may be linked together by one or more communication buses 118.

[0097] The storage medium 104 may be any form of non-volatile data storage device, such as one or more of a hard drive, a magnetic disk, a solid-state storage device, an optical disk, a ROM, etc. The storage medium 104 may store an operating system for execution by the processor 108 to operate the computer 102. The storage medium 104 may also store one or more computer programs (or software or instructions or codes).

[0098] The memory 106 may be any random access memory (storage unit or volatile storage medium) suitable for storing data and / or computer programs (or software or instructions or code).

[0099] The processor 108 may be any data processing unit suitable for executing one or more computer programs (such as those stored on the storage medium 104 and / or in the memory 106), some of which may be computer programs according to embodiments of the present invention, or computer programs that, when executed by the processor 108, cause the processor 108 to perform a method according to embodiments of the present invention and configure the system 100 as a system according to embodiments of the present invention. The processor 108 may include a single data processing unit or multiple data processing units that operate in parallel, individually, or in cooperation with each other. In performing data processing operations according to embodiments of the present invention, the processor 108 may store data to the storage medium 104 and / or the memory 106, and / or read data from the storage medium 104 and / or the memory 106.

[0100] The interface 110 can be any unit for providing an interface to a device 122 that is external to or removable from the computer 102. The device 122 can be a data storage device, such as one or more optical disks, magnetic disks, solid-state storage devices, etc. The device 122 can have processing capabilities—for example, the device can be a smart card. Thus, the interface 110 can access data from the device 122, provide data to the device 122, or interface with the device 122 based on one or more commands it receives from the processor 108.

[0101] The user input interface 114 is arranged to receive input from a user or operator of the system 100. The user may provide this input via one or more input devices of the system 100, such as a mouse (or other pointing device) 126 and / or a keyboard 124 connected to or in communication with the user input interface 114. However, it should be understood that the user may provide input to the computer 102 via one or more additional or alternative input devices, such as a touch screen. The computer 102 may store input received from the input devices via the user input interface 114 in the memory 106 for subsequent access and processing by the processor 108, or may pass it directly to the processor 108 so that the processor 108 can respond to the user input accordingly.

[0102] The user output interface 112 is arranged to provide graphical / visual and / or audio output to a user or operator of the system 100. Thus, the processor 108 may be arranged to instruct the user output interface 112 to form an image / video signal representative of the desired graphical output and to provide this signal to a monitor (or screen or display unit) 120 of the system 100 connected to the user output interface 112. Additionally or alternatively, the processor 108 may be arranged to instruct the user output interface 112 to form an audio signal representative of the desired audio output and to provide this signal to one or more speakers 121 of the system 100 connected to the user output interface 112.

[0103] Finally, network interface 116 provides computer 102 with the functionality to download data from and / or upload data to one or more data communications networks.

[0104] It should be understood that Figure 1 The architecture of the system 100 shown in FIG. 1 and described above is merely exemplary, and systems with different architectures (e.g., having a larger Figure 1 Fewer components than shown in Figure 1 116 ). As an example, the computer system 100 may include one or more of the following: a personal computer; a server computer; a tablet computer; a laptop computer; etc. Additionally, it is possible that some components of the computer system 100 are not located in a personal computer, a server system, or a laptop computer, but are instead part of a computer network connected to the personal computer, server system, or laptop computer via the network interface 116, or are located in a cloud computer network.

[0105] 2 - Example Embodiments Discussed in Connection with Vehicle Software Systems

[0106] 2.1 – Example Framework

[0107] In this section, example embodiments are discussed in the context of a vehicle software system. However, as noted above, it should be understood that the techniques and issues discussed herein are more broadly applicable to other types of software systems, and that embodiments of the present invention are not limited to software systems for (at least in part) controlling vehicle operation.

[0108] Vehicle software systems are complex systems that rely on a variety of technologies to operate and provide intelligent functionality. Gray-box fuzz testing can assess the security of software components using a wide range of input combinations. Some embodiments of the present invention provide a vulnerability-oriented fuzz testing framework (herein referred to as the "framework") that verifies the security of vehicle software systems using a wide range of valid inputs, aiming to thoroughly examine their vulnerable components. The framework directs testing towards the system's most vulnerable (or weakest) components by leveraging security vulnerability metrics specific to vehicle software system challenges. Using the system's source code, the framework automatically identifies vulnerable or vulnerable functions within the system using these metrics and assigns a weight (w) to each function based on the metric(s). Higher vulnerability scores indicate weaker components, resulting in higher w values. The framework prioritizes vulnerable functions and focuses its inspection on them. Unlike other gray-box techniques, the framework focuses not only on coverage but also on the number of times a vulnerable component is traversed (i.e., at least partially executed as part of a test). The weights assigned to functions indicate a threshold for testing. The framework can be fed with a sample of good inputs (i.e., inputs known to be valid for the software system) to generate a series of valid test cases. The framework runs each test case to monitor whether it passes through a weighted function or is connected to a weighted function. Such test cases allow vulnerable components to be validated, thus moving them to a high-priority queue for the creation of more test cases. In contrast, test cases that do not cover weak functions receive less attention.

[0109] Figure 2a and 2b The framework is shown together. Figure 2a shows the framework steps, which in some embodiments may be performed by Figure 2b The four engines shown in FIG. 1 are automatically executed: a vulnerability engine, a mutation engine, an assessment engine, and a prioritization engine. The vulnerability engine measures the vulnerability of a function. The mutation engine generates a series of valid inputs to check / test the software system. The assessment engine evaluates the usefulness of test cases. Finally, the prioritization engine prioritizes testing for more vulnerable components. However, it should be understood that embodiments of the present invention can be implemented in various ways, and the use of these four engines and the distribution of functionality across them as described above are merely examples.

[0110] Now refer to Figure 2a. The preparation of the fuzz testing routine (steps 1, 2, and 3) can be run at compile time to minimize the overhead of the security testing phase. In step 1, the framework calculates the security vulnerability value of each component using the source code of the software system and assigns weights (w) to vulnerable functions. In step 2, a call graph of the software system is generated. In step 3, by using sample inputs, the framework can build a dictionary to identify the input format of the software system. In this embodiment, two queues are used (a high priority queue and a low priority queue) - these queues can be initialized by adding these sample inputs (or initial seeds) to the high priority queue.

[0111] The remaining steps (steps 4 to 9) can be viewed as a fuzz testing routine as shown in Algorithm 1.

[0112]

[0113] This routine is initiated during the security testing phase. In step 4, the framework begins by selecting a seed input from a high-priority queue. If the high-priority queue is empty, the low-priority queue is activated. If both queues are empty, the process terminates. In step 5, the selected seed is mutated, and the software system is executed with the mutated seed as the new input. In step 6, the framework updates the coverage table (i.e., a table indicating which functions have been (at least partially) called or executed) and the call counts of the weighted functions based on the seed execution. The framework prioritizes the tests based on the results. Specifically, in step 7, if the test case passes through a vulnerable function or has a path to a vulnerable function and the call count is less than the assigned weight, the framework adds the mutated input to the high-priority queue; in step 8, if the high-priority queue requirements are not met but at least one new branch is discovered, the vulnerability-oriented fuzz testing framework adds the mutated input to the low-priority queue; and in step 9, if the conditions of both queues are not met, the mutated seed is discarded.

[0114] like Figure 2b As shown, the vulnerability engine is responsible for identifying the possibility that system functions have vulnerabilities and building a call graph.

[0115] The vulnerability engine can create a call graph at compile time because the assessment engine (discussed in more detail below) needs to direct testing to vulnerable functions. The call graph (CG) of a software system (or component (C)) has a set of nodes, where (N) represents the total number of nodes in the CG. Each node in the CG represents a function, and a directed edge (n→n*) between two nodes reveals the possibility of traversing from function n to function n*.

[0116] The second role of the vulnerability engine is achieved by employing one or more security metrics designed to identify vulnerabilities in software systems. These metrics can target the uniqueness and heterogeneity of the system to more accurately reflect its architecture and expose vulnerabilities.

[0117] A vulnerability engine can take the source code of a software system as input and automatically analyze it using one or more security metrics to identify functionality that poses a high risk to the system. If the component is outsourced, the metrics can be run at the development company. High-risk functionality is best tested thoroughly to expose system failures early.

[0118] Existing gray-box testing techniques only strive to expand code coverage without distinguishing between weak system functions. However, some functions must be checked multiple times. For example, consider the script presented in Listing 1 below:

[0119] List 1:

[0120] result=0

[0121] if x>=0:

[0122] result = 100 / x.

[0123] If x is assigned a value greater than 0, the script runs normally. However, when the value of x is 0, the script throws an exception. Therefore, insufficient coverage exposes some bugs in the software system. At the same time, testing all functions of a software system multiple times within a specific timeframe is not feasible. Security metrics guide the framework toward functions that require special treatment and intensive testing to maximize early bug disclosure. The higher a function's overall security vulnerability metric, the greater the risk it poses. Based on the security vulnerability of a function, a weight w can be assigned, representing the number of times that function must be tested.

[0124] The security vulnerability of function F in the vehicle software system can be calculated using one or more security vulnerability metrics in a variety of ways. For example, a single security vulnerability metric can be used. Alternatively, the security vulnerability of function F can be calculated as a weighted sum of multiple security vulnerability metrics, such as according to Equation 1 below. It will be appreciated that the security vulnerability of function F can be calculated in other ways.

[0125] Equation 1:

[0126]

[0127] To prioritize functions based on their vulnerability values, each parameter (ie, each value generated by a security vulnerability metric) may be divided by the maximum value achieved for the same security vulnerability metric across all functions.

[0128] ECR(F) represents the ECU coupling risk of function F. ECR measures the risk posed by ECU coupling, which could allow malicious messages to propagate from one vulnerable component to another in the system. ECR(F) is determined by counting the number of ECUs in F that are coupled to other ECUs in the system. More details on ECR, including how it can be calculated, can be found in Section III.A of

[67] .

[0129] CR(F) represents the communication risk of function F. CAVs use different communication means, exposing the vehicle to various threats

[64] . CR uses the weights of communication means defined by safety engineers based on their criticality. CR(F) can then be calculated by identifying the set of communication means employed by F. More details on CR, including how it can be calculated, can be found in Section III.B of

[67] .

[0130] CXR(F) represents the complexity risk of a function F. Complex code is challenging to develop and maintain, which increases the likelihood of vulnerabilities. CXR(F) can be defined as the combination of the number of source code lines of code (SLOC) and the nesting complexity of F. More details on CXR, including how it can be calculated, can be found in Section III.C of

[67] .

[0131] DR(F) represents the risk associated with the fluctuating inputs and outputs of a function F, which could become a window for attackers to compromise the system if not adequately tested. DR(F) can be assessed by identifying sets of fluctuating inputs, fixed inputs, fluctuating outputs, and fixed outputs. Since fluctuating inputs and outputs pose a higher risk, weights can be added to these sets. More details on DR, including how it can be calculated, can be found in Section III.D of

[67] .

[0132] HIST(F) represents the history of security issues with F. Functions that have previously facilitated successful attacks need to be re-evaluated and retested to ensure adequate security. HIST(F) can be calculated by counting the number of attacks that have previously affected F. HIST can also utilize a forgetting factor to assign higher importance to recent attacks that may not have been resolved. More details on HIST, including how HIST can be calculated, can be found in Section III.E of

[67] .

[0133] The weights of the weighted sum (i.e., α, β, γ, δ, θ in the example equation 1 above) can be set by the user based on the relative importance of the metrics perceived by the user or based on a specific goal (e.g., if the purpose of the test is to specifically check for certain types of vulnerabilities). Alternatively, the weights of the weighted sum (i.e., α, β, γ, δ, θ in the example equation 1 above) can assume respective predetermined values.

[0134] The weight w of function F, i.e., the target number of times function F is to be tested, can then be determined based on the calculated security vulnerability for function F. For example, the weight can be proportional to the calculated security vulnerability value; various possible value ranges for the security vulnerability can be set, each with an associated target number, where the weight of function F is set to the target number associated with the range that the security vulnerability value of F falls into; and so on.

[0135] For example, in one embodiment:

[0136] α=7;

[0137] β = 1;

[0138] γ = 1;

[0139] δ = 2;

[0140] θ = 4; and

[0141] The weight w can be calculated as:

[0142] i. If the calculated security vulnerability value is greater than 6, then w = 100

[0143] ii. If the calculated security vulnerability value is greater than 2.5 but less than 6, then w = 50

[0144] iii. Otherwise w = 0.

[0145] It should be understood that other weight sets and other methods of calculating the weight w can be used. For example, if the calculated security vulnerability value is less than 1, the weight w can be set to 0, otherwise it is set to a predetermined positive value.

[0146] As described above, the mutation engine can mutate seeds obtained from one of the queues to generate test seeds to be provided as input to the software system. In some embodiments, the mutation engine can also be designed to generate test seeds that pass any verification standard of the automotive components to extend code coverage. The automotive components communicate via CAN or Flexray buses. Random mutations in communication messages may cause security tests to fail at the data verification step, leaving critical parts of the code unverified. For example, AFL's mutation engine performs a small bit-level mutation on good inputs to generate a series of seed inputs. AFL is designed for compressed data formats such as multimedia files, images, and compressed data. Bit-level mutation has some key limitations when applied to systems with specific formats, such as automotive software systems

[63] . Although bit-level mutation introduces minor changes that have little impact on the input, the mutation may destroy the input structure. In addition, bit-level mutation cannot preserve the input data type. To overcome these challenges, in some embodiments, the mutation engine can adopt an input structure-aware mutation method consisting of three main components: (1) input format, (2) data type-based mutation, and (3) crossover-based mutation. The input format can be identified before starting the fuzz testing routine. The framework then passes the seed input to the mutation engine to perform data type-based mutation. After completing the fuzzing process using data type-based mutation, the mutation engine switches to crossover-based mutation to find good test cases and extend code coverage—for example, periodically performing crossover-based mutation on seeds obtained from the queue (e.g., once for every nth seed obtained from the queue for some positive integer n) instead of or in addition to data type-based mutation.

[0147] For input formats, several solutions have been proposed to reduce dropped messages and achieve mutation structure awareness, including: taint-based mutation, input parsers, and dictionaries

[68] . Taint-based fuzzers require extensive code analysis, which increases testing overhead

[69] . Input parsers employed by grey-box fuzzers are used to identify input structures to guide mutations towards data blocks and preserve necessary file headers. Nevertheless, these input parsers are best suited for processing media files, video files, and network files

[63] . Therefore, preferably, the mutation engine utilizes a dictionary to preserve the input format. Dictionaries are a robust technique widely used to provide information about the input to the fuzzer, improving the efficiency of fuzz testing

[70] . Vulnerability-oriented dictionaries mark file headers and prerequisite fields that are necessary to prevent input loss. Techniques for input format learning and compliance are well known and will not be discussed further in this paper - embodiments of the present invention can utilize any such techniques (although this is optional).

[0148] After identifying the input format, the mutation engine attempts to automatically identify the data field types. This step enables performing mutations based on data types, which helps the seed input pass the initial validation step and explore the system. This mutation technique triggers more errors than random mutation because it cleverly preserves the structure of the input while validating the system with different input ranges

[71] .

[0149] In some embodiments, for each seed input, the mutation engine performs one mutation operation on a field. Small mutations are preferably performed to preserve the bulk of the seed, which is useful for exploring the system and testing vulnerable components. The mutation engine may first attempt to parse the field to be mutated into a data type, such as numeric, Boolean, or string. Depending on the data type, a set of operations may be performed. For numeric data, the mutation engine may randomly select one of the following mathematical operations: subtraction, multiplication, division, and addition. Given a numeric field X, an arbitrary numeric field Y is generated to randomly apply one of the mathematical operations (for example, if the randomly selected operation is addition, the mutated field is X + Y). The mutation engine may mutate Boolean data to true or false, for example, the opposite of the field's current Boolean value. For strings, the mutation engine may randomly delete, insert, or flip a single bit. If the mutation engine cannot identify the data field type, it may perform a random one-bit mutation. Furthermore, to test the system's input validation routines, the mutation engine may mutate fields to different data types (for example, mutating a numeric field to a string). However, in some embodiments, this validation is performed only once for each field to avoid stalling during validation and exploring the system.

[0150] As mentioned above, crossover-based mutation can also be used. Several grey-box fuzzers are known to use this type of mutation,

[63] ,

[72] . Some embodiments involve statically swapping chunks of different seeds to preserve the input structure. Given a seed s, this may involve randomly selecting a portion p, where p1 and p2 are the start and end indices of the portion. Another portion p* is cut out from the random seed s* using the same indices. The portion p is then placed in the place of p* in s*, and p* is placed in the place of p in s, thus generating two new seeds. The positions of the swapped portions are preserved to maintain the format of the seed.

[0151] Seed mutation techniques are well known and will not be discussed further here - embodiments of the present invention may utilize any such techniques.

[0152] The framework can be guided towards vulnerable components and coverage extensions. The evaluation engine helps achieve this by monitoring the performance of seed inputs.

[0153] For each test seed input to the software system for testing, the evaluation engine can record the edges of the call graph that are traversed. It can use lightweight instrumentation to detect branch coverage. Branch coverage provides much more insight into the execution path than statement coverage. It can identify branches with conditional statements that cannot be identified with simple statement coverage

[73] . Coverage helps the fuzz tester understand the system state and identify the usefulness of the test seed input.

[0154] In order to successfully guide the fuzz tester towards vulnerable components, the evaluation engine can detect seed inputs that pass through vulnerable functions or have paths leading to (or close to) vulnerable functions. Using weighted functions created by the vulnerability engine, the evaluation engine identifies vulnerable functions and monitors test cases that pass through them. The framework attaches great importance to vulnerable functions and strives to thoroughly verify their security. Therefore, even if the seed input is not passing through the vulnerable function, the evaluation engine will check whether the seed input can eventually reach the vulnerable function. Inputs that pass through nodes connected to vulnerable functions have the opportunity to reach the vulnerability through slight mutations. The call graph generated by the vulnerability engine can be used to determine whether the executed input has a path that can reach the vulnerable function other than the system entry point. Figure 3 An example call graph is shown. Given Figure 3 Consider a call graph with a vulnerable function n7. A seed input has a path to n7 only if it passes through nodes n3 or n6. For example, consider a seed input s1 that spans nodes n1, n2, and n4. It's unlikely that seed s1 will reach node n7. Therefore, it can be marked as unhelpful for testing vulnerable functions.

[0155] In large and complex systems such as vehicle software systems, test case prioritization is critical during the testing and validation phase. Vulnerabilities in systems are increasing under limited time budgets. Existing grey-box fuzz testing techniques do not differentiate between test cases, and they all reside in the same queue and are executed in a first-in, first-out (FIFO) order. In contrast, embodiments of the present invention prioritize test cases based on the following findings from the test cases: seeds that trigger vulnerable functions are given high priority. The prioritization engine can analyze the coverage table and weighted function counts generated by the evaluation engine to determine whether the seed input should be added to the high priority queue, the low priority queue, or discarded. More than two queues can be used if the safety engineer needs to target functions at multiple thresholds.

[0156] As discussed in the Vulnerability Engine, each identified vulnerable function is assigned a weight (w) to thoroughly test the vulnerable function. Test cases that explore vulnerable functions or have paths leading to vulnerable functions and whose counts are less than the assigned weight are very useful and are therefore added to the high priority queue. Test cases that do not exercise vulnerable functions and have no paths leading to vulnerable functions but extend code coverage (i.e., discover new branches not discovered earlier) are considered lower priority and moved to the low priority queue. Conversely, test cases that do not explore new branches and do not exercise (or approach) vulnerable functions are not added to any queue.

[0157] Seed inputs that are queued can be assigned an "energy value" to be further mutated and used as new inputs in the fuzz testing routine. The energy value represents the number of times the seed input has been mutated (i.e., the number of times the seed will be used to generate further mutated seeds for each individual test). The prioritization engine uses a constant energy allocation while giving more energy to seeds that explore vulnerable components. High-priority seed inputs that pass through vulnerable components are given three times the energy of low-priority seed inputs, allowing them to generate more inputs, providing a better chance of exploring vulnerable components. Seeds that belong to the high-priority queue but do not pass through vulnerable components are assigned twice the energy of low-priority seeds. Such test cases have a high probability of passing through vulnerable components, but they may never reach them.

[0158] For example, consider Figure 3 , where the execution count of the vulnerable node n7 (i.e., the number of times n7 has been tested) is less than its weight. A seed s that passes through n7 can be assigned an energy value of 3x, where x is a constant defined by the security engineer. A seed s* that executes nodes n1 and n3 (but not n7) is assigned an energy value of 2x. Therefore, in order to save fuzz testing power for weak functions, test cases similar to s* are assigned less energy values ​​than test cases similar to s that guarantee vulnerability exploration. On the other hand, test cases belonging to the low-priority queue are assigned lower energy values. The seed s** that first discovered the edge n1→n2 was assigned an energy value of x.

[0159] 2.2 – Evaluation of the Example Framework

[0160] To evaluate the efficiency and performance of the framework presented in Section 2.1 above, we apply it to the automotive system OpenPilot as an example and compare the framework with two other fuzz testing methods: AFL and a mutation-based fuzzer.

[0161] OpenPilot is an open-source driving and safety assistance system developed by comma.ai. It provides SAE Level 2 driver assistance features, enabling adaptive cruise control (ACC), automatic lane centering (ALC), forward collision warning (FCW), and lane departure warning (LDW). It supports various car models, including those from Honda, Toyota, Hyundai, and Lexus. The system also provides safety features by implementing a driver monitoring (DM) function that warns inattentive drivers.

[0162] Safety-critical systems like these require intensive safety testing to verify and validate their robustness against malicious behavior. Fuzz testing generates a large number of unexpected inputs that could trigger inappropriate behavior in the system. OpenPilot supports the regression testing tool Process Replay, which simulates system processes and verifies output against predefined inputs. To run fuzz tests, the tool is tuned to accept a variety of inputs. To verify the efficiency of vulnerability-oriented fuzz testing frameworks, comparisons were conducted with the fuzz tester American Fuzzy Lop (AFL) and an unguided mutational fuzzer. OpenPilot was designed using Python and C. Since the original AFL did not support Python, a Python derivative of AFL was used with some adjustments that did not affect AFL's behavior and main functionality but enabled it to understand OpenPilot processes. To compare the efficiency of gray-box fuzz testing with black-box fuzz testing in automotive systems, an unguided mutational fuzzer was designed.

[0163] An embodiment of the framework was built using Python. All experiments were performed on a single machine equipped with an Intel Core i7-1065G7 processor, a quad-core hyperthreaded chip running at a base frequency of 1.3 GHz, and 8 GB of memory. This machine ran 64-bit Ubuntu 16.04 Long Term Support (LTS) system.

[0164] To obtain the results, both the framework and AFL were executed until they could no longer discover new branches or reach vulnerable functionality. Then, an unguided mutational fuzzer was run against the same number of test cases generated by the framework. To test the efficiency of the framework, four different comparisons were performed: the number of test cases, the number of dropped messages, the number of coverage, and the number of crashes.

[0165] 1) Test case analysis

[0166] As shown in Table 1 below, the framework generated 1,810 test cases, 808 more than AFL. The number of test cases affects processing time. AFL completed execution in half the time of the other two fuzzers. As mentioned above, the framework assigns weights to vulnerable functions for multiple verification. Therefore, even if a test case does not extend coverage but instead evaluates a vulnerable function, it remains in the queue and is further mutated. In contrast, AFL only stores test cases that extend coverage. As a result, AFL requires fewer test cases to achieve its goals.

[0167] Table 1:

[0168]

[0169] 2) Abandoned test case analysis

[0170] The efficiency of the mutation engine can be examined by looking at the number of discarded messages generated by each test harness. As mentioned above, the mutation engine may attempt to mutate inputs with incompatible data types to verify the system's input validation routines. As a result, the framework generated 20 discarded messages. AFL's mutation engine has significantly more discarded messages than both this framework and the unguided mutation fuzzer. Specifically, out of the 1,002 test cases generated by AFL, 233 test cases failed OpenPilot's input validation routines. This represents 23% of the test cases, compared to 1% for the other two test harnesses. Automotive systems like OpenPilot have a rigorous verification scheme, making random mutation an ineffective method for verifying system safety.

[0171] For example, Figure 4 An overview of sample inputs for OpenPilot is provided. To determine if a vehicle is healthy, the system takes as input a voltage value, an ignition wire boolean value, an allowed control boolean value, a CAN send error value, and a CAN forward error value. The seed s represents a good input for the mutation engine to generate new seeds. The framework's mutation engine performs small mutations based on the input fields, resulting in two new inputs S1 and S2 that meet the criteria and help validate the system. The AFL mutation engine performs a one-bit mutation, changing the "A" in "FALSE" to "@" in S3 and the "0" to "p" in S4. Both of these new inputs S3 and S4 do not meet OpenPilot's input validation process and are discarded.

[0172] AFL wasted approximately 1.8 hours of processing time on invalid inputs. Therefore, the mutation engine of some embodiments of the framework outperforms small random mutation strategies and focuses on testing valid inputs that can explore the code and find vulnerabilities.

[0173] 3) Coverage analysis

[0174] Table 1 shows the total number of conditional branches visited. Branch coverage for all three methods is relatively similar, reaching approximately 91% of the system's conditional branches. The framework hits 3 more branches than AFL and 12 more branches than the unguided mutation fuzzer. Because the framework and AFL implement the same strategy for expanding code coverage, they typically share similar coverage results. Due to the weights assigned to vulnerable functions, the framework achieves slightly better branch coverage. Mutating test cases that fail to find new branches but thoroughly validate weak components ultimately generates seed inputs that discover new branches.

[0175] By analyzing the impact of weights on coverage behavior, we can further explore the coverage of the testing tool. Figure 5 Statement coverage curves for each testing tool were plotted. Statement coverage was used in this analysis because it provides a broad view of coverage. AFL reached optimal coverage in 6 hours, while the framework required 15.5 hours. During the initial 15.5 hours, the framework prioritized search and evaluation toward vulnerable components rather than coverage expansion. Once comprehensive testing of high-priority functionality was complete, the fuzz tester switched to low-priority testing. The primary goal of this phase was to expand coverage, which was achieved quickly by the framework because test cases that contributed to coverage expansion were kept in a low-priority queue and not neglected.

[0176] While AFL's coverage graphs and those of the frameworks have similar shapes, the unguided mutational fuzzer exhibits a different pattern. While the coverage of the other tools increases dramatically, the fuzzer reaches its optimal coverage gradually. This difference highlights the importance of test guidance. The unguided mutational fuzzer attempts to randomly verify the system. Without knowledge of test performance, the fuzzer cannot identify anomalous test cases that pass through the system. After wasting over 11 hours looping around the same functionality, the fuzzer randomly hits more statements.

[0177] 4) Collision analysis

[0178] Figure 6 This figure depicts crashes triggered by three test tools. A crash is an anomaly caused by unexpected behavior in the vehicle system. Most of the crashes detected by the test tools are index out-of-bounds exceptions. For example, the software system expects the radiator fan speed to be between 0 and 65535 RPM. Any value greater than that will cause the system to crash.

[0179] As shown in the figure, the number of crashes identified by the framework exceeds the number of crashes identified by AFL and unguided mutation fuzzers. The framework detected a total of 335 crashes. Figure 6The plot shows an exponential increase in the number of crashes found by the framework. Consistently, the framework found crashes within the first 15.5 hours of testing. At that time, the fuzzer was sticking to the test cases that were weighted for functionality. This is reflected in Figure 5 , where a stable coverage map provides a thorough assessment of weak features.

[0180] The unguided mutational fuzzer detected a total of 176 crashes. The mutation engine and the number of generated test cases improved the testing tool's performance, enabling it to find more crashes than AFL. The random fuzzer intentionally ran 1,810 test cases to assess the importance of gray-box testing in the automotive industry. This gave the fuzzer a fair chance to find crashes. Despite this, the framework found 90% more crashes than this black-box testing approach. The effectiveness of the mutation engine undoubtedly improved the performance of black-box verification. The fuzzer did not waste time on invalid inputs; 99% of test runs were successful. Random black-box fuzzing techniques, which attempt to create arbitrary inputs not accepted by automotive systems, yielded less effective results.

[0181] AFL performed poorly in finding crashes. AFL detected eight crashes in the first 4.5 hours. As mentioned earlier, AFL's mutation engine works well on media files. However, it is less effective against complex systems that include robust input validation mechanisms. Testing time is wasted on invalid inputs that do not evaluate the system and seek crash identification. AFL reaches its coverage peak relatively quickly. However, this affects the number of crashes detected. Figure 5 As shown, during the first 4.5 hours, the fuzzer was still trying to expand coverage, but was hitting vulnerable functions. Once AFL increased its coverage, it found fewer crashes.

[0182] The relationship between weighted functions and collapse can be further investigated. Figure 7 The chart compares the number of crashes detected with the number of times the three testing tools tested weak components. The framework uses security vulnerability metrics to identify weak components of the system. Weights are assigned to these components to ensure a thorough evaluation. The framework checked weak components at least 808 times, compared to 188 times for the unguided mutation fuzzer and 79 times for AFL. Figure 7 As shown in the chart, the more vulnerable components tested, the more crashes discovered. AFL's weighted function executions are low, which is reflected in the number of crashes discovered. Conversely, the framework's exhaustive evaluation of vulnerable components enhances its crash detection capabilities. This demonstrates the importance of safety metrics and weight distribution. Safety metrics direct testing to complex functions that are more prone to errors. Weight distribution gives the framework the opportunity to examine these components more frequently and identify vulnerabilities.

[0183] Figure 8 The Venn diagram depicts the similarities between crashes reported by the three testing tools. The framework identified all crashes identified by AFL and 153 unexpected behaviors found by the unguided mutation-based fuzzer. Not only did the framework identify 15 of the crashes found by the mutation-based fuzzer, representing 4% of the total, but it also found 90% more crashes.

[0184] 2.3 – Conclusion of the Example Framework

[0185] Building a vehicle that can safely and reliably drive, perceive its surroundings, and entertain its passengers requires integrating approximately 100 million lines of code, dozens of electronic devices, and multiple advanced technologies into a single system, exposing the vehicle to numerous potential cyberattacks. Static code analysis, dynamic program analysis, vulnerability scanning, penetration testing, and fuzz testing are security assurance methods that help OEMs and suppliers ensure system security during vehicle software engineering (VSE). Despite this, the automotive industry faces several challenges that continue to make security testing a daunting task. These challenges include system complexity and scale, outsourcing, input and output fluctuations, and testbed complexity.

[0186] Black-box fuzz testing is a tool proposed to mitigate these challenges. However, its naive nature makes it an unreliable testing tool, leaving critical systems with minimal security resilience. White-box fuzz testing offers a more reliable security testing tool. However, given the scale of the system, white-box testing becomes a time-consuming task that is difficult to manage within strict project deadlines.

[0187] The vulnerability-oriented gray-box fuzz testing framework discussed above overcomes the limitations of black-box testing by acquiring some knowledge about the system without incurring the overhead of white-box testing. Unlike black-box fuzz testers that blindly check the system, this framework leverages safety metrics to supervise and guide testing. Safety metrics quantitatively measure the vulnerabilities of components within the vehicle software system. This estimate can reflect code complexity and identify weak integrations that attackers could exploit. Based on the vulnerability value, each component is assigned a weight representing the number of times it should be tested. Thoroughly checking weak functionality promotes vulnerability detection and ensures system safety. The framework monitors seed input coverage to achieve its goals and prioritizes testing. To enhance the performance of the gray-box fuzz tester, the mutation engine can be configured to generate diverse test cases that conform to the input structure of the automotive system by inferring the input data type.

[0188] As can be seen, the framework provides a reliable security testing tool that does not increase testing complexity, but instead intelligently and effectively identifies weak functions to focus on them. In addition, prioritizing tests can help security engineers automatically manage security tests in time-constrained projects.

[0189] 3-Overall Discussion

[0190] More generally, an embodiment of the present invention provides a method for fuzz testing a software system, wherein the software system includes a plurality of callable units and is arranged to receive input for processing by the software system, the method comprising: determining, for each callable unit in the plurality of callable units, a target number (or quantity) of times (or quantities) to test the callable unit based on one or more security vulnerability metrics; initializing a plurality of sorted queues, each queue being configured to store one or more seeds, the initialization comprising storing the one or more initial seeds in a corresponding queue in the sorted plurality of queues; and executing a series of tests, wherein executing each test comprises:

[0191] Get the seed from the highest sorted non-empty queue;

[0192] performing a mutation process on the obtained seeds to generate test seeds;

[0193] providing a test seed as input to the software system for processing by the software system; and

[0194] evaluating the software system's processing of test seeds to generate test results;

[0195] wherein each queue in the ordered plurality of queues has an associated seed addition criterion, and wherein performing each test comprises: (a) adding a test seed to a highest-ranked queue in the ordered plurality of queues for which the test seed satisfies the seed addition criterion associated with that queue; or (b) discarding the test seed if the test seed does not satisfy the seed addition criterion associated with any queue in the ordered plurality of queues;

[0196] The seed addition criteria are configured such that: if the software system's processing of a first test seed involves the execution of a callable unit of interest or an execution path close to the callable unit of interest and if the software system's processing of a second test seed does not involve the execution of a callable unit of interest or an execution path close to the callable unit of interest, the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added, wherein if the current number of tests that result in the execution of a callable unit is less than the target number of times the callable unit is to be tested, the callable unit is a callable unit of interest.

[0197] Figure 11The example use of the embodiment of the present invention is schematically shown. As mentioned above, the embodiment of the present invention relates to testing a software system, such as testing for vulnerabilities, bugs, errors, etc. The software system to be tested is Figure 11 1 is shown as system 1100. In the example discussed in Section 2 above, system 1100 includes a software system (or portion thereof) of a vehicle or for controlling a vehicle - however, it should be understood that system 1100 can be used to perform other functions and / or for other situations / configurations. Software system 1100 includes a plurality of "callable units" 1102 - here, each callable unit 1102 can be a respective one of the following: a routine; a subroutine; a function; a procedure; a process; a class method; an interface; a component; or a subsystem of a larger system. Callable units 1102 can, for example, be stored in or as one or more files (e.g., as source code and / or as compiled executable instructions). In the example discussed in Section 2 above, one or more (possibly all) callable units 1102 are one of the following: an ECU, an integrated ECU (multi-function computer), and a process. Software system 1100 can be intended to be implemented on a hardware system 1104 (e.g., one or more processors of a vehicle, as discussed in Section 2 above; Figure 1 The system 1100 may be executed on / by one or more computer systems 1100 of the hardware system 1106, as discussed above; etc. The system 1100 may be arranged to receive one or more inputs 1106, i.e., data to be processed. For example, the system 1100 may expose one or more interfaces for receiving input data, e.g., in the form of one or more of: signals from sensors; messages from other systems / components; indications of events that have or have not occurred; data from one or more data sources (e.g., databases, web pages, etc.); time and / or date data from a clock; etc. Additionally or alternatively, the system 1100 may be arranged to receive responses to queries issued by the system 1100 (e.g., queries issued to a server or other device / component). One or more of the inputs 1106 may be received / obtained from a source external to the hardware system 1106; additionally or alternatively, one or more of the inputs 1106 may be received / obtained from a source internal to the hardware system 1106 (e.g., a clock of the system 1106).

[0198] Embodiments of the present invention relate to performing Figure 11A method for fuzz testing a software system such as the system 1100 of FIG. 1 is provided. Such a method may be implemented by a test system 1110. The test system 1110 may, for example, include one or more computer systems 100. The test system 1110 may be arranged to communicate / interact with the software system 1100 via a network 1120 (although it will be appreciated that the test system 1110 may be directly coupled to the system 1100 or may communicate with the system 1100 in other ways). Alternatively, the fuzz testing may be implemented by the hardware system 1104 itself (such that the test system 1110 and the hardware system 1104 are the same system). It will be appreciated that other configurations and architectures for performing fuzz testing are also possible.

[0199] In summary, the test system 1110 performs fuzz testing by simulating or providing test inputs 1106 for processing by the software system 1100. The results of this processing (which may simply be an indication of whether the software system 1100 crashed or otherwise failed or exhibited a malfunction) may be obtained / monitored by the test system 1110, which then helps guide the formation of subsequent test inputs 1106 for processing by the software system 1100—the goal being to generate test inputs such that testing is targeted or biased toward certain portions of the software system 1100 (i.e., the test inputs are generated to ensure that those specific portions of the software system 1100 are executed more frequently as part of the test than other portions of the software system 1100).

[0200] Figure 9 is a flow chart illustrating a method 900 according to some embodiments of the present invention. The method 900 may be, for example, Figure 11 110. Specific examples of method 900 have been discussed above in Section 2 with reference to the "Framework." The example embodiment ("Framework") set forth in Section 2 above is described with reference to a test vehicle software system. However, it should be understood that the techniques and issues discussed herein are more broadly applicable to other types of software systems 1100, and that the embodiments of the invention herein should not be considered limited to software systems for (at least in part) controlling vehicle operation. In the example embodiment ("Framework") set forth in Section 2, the seeding criterion for the high priority queue is that the processing of the test seed by the software system 1100 involves the execution of the callable unit of interest or is close to the execution path of the callable unit of interest. Similarly, the seeding criterion for the low priority queue is that the processing of the test seed by the software system 1100 reaches a branch point in the software system 1100 that has not been reached when executing previous tests. However, it should be understood that other and / or alternative seeding criteria may be used. For example:

[0201] The seed addition criteria may be configured such that: if processing of a first test seed by the software system 1100 involves an execution path that approaches a callable unit of interest but does not involve execution of the callable unit of interest, and if processing of a second test seed by the software system 1100 involves execution of the callable unit of interest, the queue to which the first test seed is added has a higher rank than the queue to which the second test seed is added. Thus, testing is directed or biased toward using seeds that nearly (but so far have failed) to reach the callable unit of interest, rather than seeds that actually do reach the callable unit of interest.

[0202] The seed addition criteria can be configured such that: if the processing of a first test seed by the software system 1100 involves an execution path close to a callable unit of interest but does not involve execution of the callable unit of interest, and if the processing of a second test seed by the software system 1100 involves execution of the callable unit of interest, the queue to which the first test seed is added has a lower ranking than the queue to which the second test seed is added. Thus, testing is directed or biased toward using seeds that have reached the callable unit of interest rather than seeds that have nearly (but so far failed to) reach the callable unit of interest.

[0203] The seed addition criteria may be configured such that: if processing of a first test seed by the software system 1100 involves execution of one or more first callable units of interest or near an execution path of the one or more first callable units of interest and if processing of a second test seed by the software system 1100 involves execution of one or more second callable units of interest or near an execution path of the one or more second callable units of interest, the first test seed is added to a queue with a higher rank than the second test seed is added to a queue if: (a) the remaining count (or number) of at least one of the one or more first callable units of interest to be tested is greater than the remaining count (or number) of each of the one or more second callable units of interest to be tested; or (b) the sum of the remaining counts (or numbers) of each of the one or more first callable units of interest to be tested is greater than the sum of the remaining counts (or numbers) of each of the one or more second callable units of interest to be tested. In this way, testing is directed or biased toward using seeds that reach callable units of interest for which more testing still needs to be performed than other callable units of interest.

[0204] Generally speaking, next, the seeds are added with standard C k(1≤k≤Z) is configured such that if the software system's processing of the first test seed involves execution of a callable unit of interest or is close to an execution path of the callable unit of interest and if the software system's processing of the second test seed does not involve execution of a callable unit of interest or is close to an execution path of the callable unit of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added. As described above, this can be achieved through various combinations and various numbers of seed addition criteria.

[0205] At step 916, a determination is made as to whether another test 950 should be performed. If another test 950 is to be performed, processing returns to step 906 so that another test 950 can be performed; otherwise, processing continues at step 918.

[0206] In some embodiments, such as the example embodiment described above (the "Framework"), testing continues until both the high priority queue and the low priority queue are empty. However, it should be understood that other criteria for terminating a test may be used instead. Thus, in some embodiments, a test sequence is executed until a termination condition is met, which is checked at step 916. For example, the termination condition may include one or more of the following: (a) each of the ordered plurality of queues is empty (e.g., as discussed above with respect to the "Framework"); (b) a threshold number of tests have been executed (which may help to end the test within a time constraint); and (c) a threshold amount of time has been spent executing the test sequence (which again may help to end the test within a time constraint). If the termination condition is met, processing may continue at step 918; otherwise, processing may return to step 906 so that another test 950 may be performed.

[0207] At step 918, the test system 1110 may perform various "end of test" processing. For example, in some embodiments of the invention, the test system 1110 may provide fuzz testing output (e.g., an indication of whether or how many crashes / bugs / errors / etc. were detected by the test, possibly along with associated metadata, as described above) based on the results / evaluations generated from the executed test 950 at step 918. Various outputs may be provided, examples of which are set forth above in Section 2.3.

[0208] As discussed above with respect to the example embodiment ("framework"), some embodiments of the present invention may utilize an "energy value"; other embodiments may not. For embodiments that do not utilize an "energy value," obtaining a seed from the highest-ranked non-empty queue at step 906 includes removing the seed from the highest-ranked non-empty queue—e.g., the queue acts as a FIFO and the seed is added to the queue only once.

[0209] However, alternatively, some embodiments of the present invention may include determining a corresponding reuse quantity for a test seed, which indicates the number of future tests (i.e., energy value) that the seed can be used as the obtained seed. This can be implemented in various ways. For example, determining a corresponding reuse quantity for a test seed may include: if the processing of the test seed by the software system 1100 (during test 950) involves the execution of a callable unit of interest, setting the reuse quantity to a first predetermined value; if the processing of the test seed by the software system 1100 (during test 950) does not involve the execution of the callable unit of interest but involves an execution path close to the callable unit of interest, setting the reuse quantity to a second predetermined value; if the processing of the test seed by the software system 1100 (during test 950) does not involve the execution of the callable unit of interest or an execution path close to the callable unit of interest but reaches a branch point in the software system 1100 that was not reached when executing the previous test, setting the reuse quantity to a third predetermined value. This may involve the first predetermined value being greater than the second predetermined value, and the second predetermined value being greater than the third predetermined value. For example, as described above with respect to the "Framework," for some positive integer x, the first predetermined value may be 3x, the second predetermined value may be 2x, and the third predetermined value may be x—however, it will be appreciated that other configurations of these predetermined values ​​may be used. Alternatively, the second predetermined value may be greater than the first predetermined value, and the first predetermined value may be greater than the third predetermined value. It will also be appreciated that different levels of energy values ​​may be associated with the test seed, and this may be based on one or more additional or alternative criteria (e.g., other factors determined when evaluating the processing of the test seed in step 912).

[0210] The test system 1110 can be implemented in various ways to achieve the "energy value". For example, some embodiments may store a corresponding reuse number for each stored seed, such that (at step 906) obtaining a seed from the highest-ordered non-empty queue includes decrementing the reuse number corresponding to the seed and (a) retaining the seed in the highest-ordered non-empty queue if the reuse number corresponding to the seed is not zero, and (b) removing the seed from the highest-ordered non-empty queue if the reuse number corresponding to the seed is zero. Alternatively, in some embodiments, (at step 914) adding a test seed to the highest-ordered queue of the sorted plurality of queues in which the test seed meets the seed addition criteria associated with the queue includes adding the test seed to the highest-ordered queue of the sorted plurality of queues in which the test seed meets the seed addition criteria associated with the queue a number of times (or a quantity) equal to the reuse number, and (at step 906) obtaining a seed from the highest-ordered non-empty queue includes removing the seed from the highest-ordered non-empty queue. Both of these approaches will result in a test seed with a reuse (energy) value of, for example, 4 being used 4 times (or one instance of the seed being used 4 times before being removed from the queue, or 4 instances of the seed, each of which is used only once before being removed from the queue). Of course, it will be appreciated that other approaches to achieving this energy functionality may alternatively be implemented.

[0211] In some embodiments, one or more of the queues may have seed addition criteria based on the number of reuses of the seed. For example, a queue may have seed addition criteria that indicates that only seeds with a reuse number above a corresponding threshold may be added to the queue.

[0212] 4 – References

[0213] The following materials are cited in the above description. The entire disclosures of these materials are incorporated herein by reference in their entirety.

[0214] 【1】S. Parkinson, P. Ward, K. Wilson, and J. Miller, “Cyber ​​threats facing autonomous and connected vehicles: Future challenges”, IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 11, pp. 2898–2915, 2017.

[0215] 【4】SVESS Committee et al., “Sae j3061 - Cybersecurity guidebook for cyber-physical automotive systems”, SAE - Society of Automotive Engineers, 2016.

[0216] 【6】L.J. Moukahal, M.A. Elsayed, and M. Zulkernine, “Vehicle software engineering (vse): Research and practice”, IEEE Transactions on Internet of Things, vol. 7, no. 10, pp. 10137–10149, 2020.

[0217]

[10] D.Oka, "Securing the modern vehicle: A study of automotive industry cybersecurity practices", February 2019;

[0218]

[11] T. Brennich and M. Moser, “Putting automotive security to the test”, ATZ electronics worldwide, vol. 15, no. 1, pp. 46–51, 2020.

[0219]

[12] B. Potter and G. McGraw, “Software security testing”, IEEE Security and Privacy, vol. 2, no. 5, pp. 81–85, 2004.

[0220]

[13] U. Drolia, Z. Wang, Y. Pant, and R. Mangharam, “Autoplug: An automotive test-bed for electronic controller unit testing and verification,” in 2011 14th IEEE International Conference on Intelligent Transportation Systems (ITSC), 2011, pp. 1187–1192.

[0221]

[15] S. Bayer, T. Enderle, D.-K. Oka and M. Wolf, “Security crash test-practical security evaluations of automotive onboard IT components”, Automotive Safety and Protection, 2015.

[0222]

[16] D. S. Fowler, J. Bryans, S. A. Shaikh, and P. Wooderson, “Fuzz testing for automotive cyber-security,” in 2018 48th Annual IEEE / IFIP International Symposium on Dependable Systems and Networks (DSN-W). IEEE, 2018, pp. 239–246.

[0223]

[19] D. K. Oka, A. Yvard, S. Bayer, and T. Kreuzinger, “Enabling cyber security testing of automotive electronics by adding monitoring capabilities,” in Embedded Automotive Security Conference, 15th European Escar, 2016, pp. 1–13.

[0224]

[20] D. S. Fowler, J. Bryans, M. Cheah, P. Wooderson, and S. A. Haikh, “A method for constructing automotive cybersecurity tests, a can fuzz testing example,” in 2019 IEEE 19th International Conference on Software Quality, Reliability, and Security Guidelines (QRS-C). IEEE, 2019, pp. 1–8.

[0225]

[21] A. Pretschner, M. Broy, I.H. Kruger, and T. Stauner, “Software engineering for automotive systems: A roadmap”, Future of Software Engineering (FOSE’07). IEEE, 2007, pp. 55–71.

[0226]

[23] M. Broy, I.H. Kruger, A. Pretschner, and C. Salzmann, “Engineering automotive software”, Proceedings of the IEEE, vol. 95, no. 2, pp. 356–373, 2007.

[0227]

[24] D. Durisic, M. Nilsson, M. Staron, and J. Hansson, “Measuring the impact of changes to the complexity and coupling properties of automotive software systems”, Journal of Systems and Software, vol. 86, no. 5, pp. 1275–1293, 2013.

[0228]

[25] S. Moshtari, A. Sami, and M. Azimi, “Using complexity metrics to improve software security,” Computer Fraud and Security, vol. 2013, no. 5, pp. 8–17, 2013.

[0229]

[26] Y. Shin and L. Williams, “Can traditional fault prediction models be used for vulnerability prediction?”, Empirical Software Engineering, vol. 18, no. 1, pp. 25–59, 2013.

[0230]

[27] I. Chowdhury and M. Zulkernine, “Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities,” Journal of System Architecture, vol. 57, no. 3, pp. 294–313, 2011.

[0231]

[28] M. Broy, “Challenges in automotive software engineering”, in Proceedings of the 28th ACM International Conference on Software Engineering, 2006, pp. 33–42.

[0232]

[29] SA Haider, G. Samdani, M. Ali, and M. Kamran, “A comparative analysis of in-house and outsourced development in software industry”, International Journal of Computer Applications, vol. 141, no. 3, pp. 18–22, 2016.

[0233]

[30] C. Hubmann, M. Becker, D. Althoff, D. Lenz, and C. Stiller, “Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles,” in 2017 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2017, pp. 1671–1678.

[0234]

[31] P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation”, SAE International Journal of Traffic Safety, vol. 4, no. 1, pp. 15–24, 2016.

[0235]

[37] J. De Winter, P. M. van Leeuwen and R. Happee, “Advantages and disadvantages of driving simulators: A discussion”, Proceedings on Measuring Behavior, vol. 2012. Citeseer, 2012, p. 8.

[0236]

[38] C. Obermaier, R. Riebl, C. Facchi, A. Al-Bayatti, and S. Khan, “Limitations of hi1 test architectures for car2x communication devices and applications,” in ACM Proceedings on Computer Science in Automotive Engineering, 2019, pp. 1–9.

[0237]

[39] I. Pekaric, C. Sauerwein, and M. Felderer, “Applying security testing techniques to automotive engineering,” in Proceedings of the 14th International Conference on Availability, Reliability, and Security, 2019, pp. 1–10.

[0238]

[40] A. Imparato, R.R. Maietta, S. Scala, and V. Vacca, “A comparative study of static analysis tools for autosar automotive software components development,” in Proceedings of the 2017 IEEE International Symposium on Software Reliability Engineering (ISSREW). IEEE, 2017, pp. 65–68.

[0239]

[45] S. Keul, “Tuning static data race analysis for automotive control software”, in 2011 IEEE 11th International Working Conference on Source Code Analysis and Processing. IEEE, 2011, pp. 45–54.

[0240]

[46] AG Bardas et al., “Static code analysis”, Romanian Economic and Business Review, Vol. 4, No. 2, pp. 99–107, 2010.

[0241]

[47] Z. B. Celik, E. Fernandes, E. Pauley, G. Tan, and P. McDaniel, “Program analysis of commodity IoT applications for security and privacy: Challenges and opportunities”, ACM Computing Research (CSUR), vol. 52, no. 4, pp. 1–30, 2019.

[0242]

[48] KA Koscher, “Securing embedded systems: analyses of modern automotive systems and enabling near-real time dynamic analysis”, PhD dissertation, 2014.

[0243]

[49] G. Cabodi, D.F.Finocchiaro and D. Montisci, “Security-oriented dynamic code analysis in automotive embedded systems”, Politecnico di Torino, Ingegneria Informatica, Corso di Laurea (Computer Engineering), 2018.

[0244]

[51] M. Ring, J. Dürrwang, F. Sommer, and R. Kriesten, “Survey on vehicular attacks-building a vulnerability database,” in 2015 IEEE International Conference on Vehicular Electronics and Security (ICVES). IEEE, 2015, pp. 208–212.

[0245]

[52] F. Sommer, J. Dürrwang, and R. Kriesten, “Survey and classification of automotive security attacks,” Information, vol. 10, no. 4, p. 148, 2019.

[0246]

[53] K. Koscher, A. Czeskis, F. Roesner, S. Patel, T. Kohno, S. Checkoway, D. McCoy, B. Kantor, D. Anderson, H. Shacham, et al., “Experimental security analysis of a modern automobile,” in IEEE Transactions on Security and Privacy, 2010, pp. 447–462.

[0247]

[54] M. Cheah, S. A. Shaikh, O. Haas, and A. Ruddle, “Towards a systematic security evaluation of the automotive bluetooth interface,” Vehicular Communications, vol. 9, pp. 8–18, 2017.

[0248]

[55] C. Corbett, T. Basic, T. Lukaseder, and F. Kargl, “A testing framework architecture for automotive intrusion detection systems”, Automotive Security 2017: Security and Reliability of Automotive Information Technology, 2017.

[0249]

[56] A. Taylor, S. Leblanc, and N. Japkowicz, “Probing the limits of anomaly detectors for automobiles with a cyberattack framework,” IEEE Intelligent Systems, vol. 33, no. 2, pp. 54–62, 2018.

[0250]

[57] T. Huang, J. Zhou, and A. Bytes, “Atg: An attack traffic generation tool for security testing of in-vehicle can bus”, in Proceedings of the 13th International Conference on Availability, Reliability and Security, 2018, pp. 1–6.

[0251]

[58] H. Liang, X. Pei, X. Jia, W. Shen, and J. Zhang, “Fuzzing: State of the art,” IEEE Transactions on Reliability, vol. 67, no. 3, pp. 1199–1218, 2018.

[0252]

[59] D. K. Oka, T. Fujikura and R. Kurachi, “Shift left: Fuzzing earlier in the automotive software development lifecycle using hil systems”, 16th escaro European Conference, 2018.

[0253]

[60] D. S. Fowler, J. Bryans, S. A. Shaikh, and P. Wooderson, “Fuzz testing for automotive cyber-security,” in 2018 48th Annual IEEE / IFIP International Conference on Dependable Systems and Networks Workshop (DSN-W), 2018, pp. 239–246.

[0254]

[61] P. Wang and X. Zhou, "Sok: The progress, challenges, and perspectives of directed greybox fuzzing", arXiv preprint arXiv:2005.11907, 2020;

[0255]

[63] V.-T.Pham,M. A.E.Santosa, A.C. Caciulescu, and A. Roy-choudhury, “Smart greybox fuzzing,” IEEE Transactions on Software Engineering, 2019;

[0256]

[64] C. Lemieux and K. Sen, “Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage,” in Proceedings of the 33rd ACM / IEEE International Conference on Automated Software Engineering, 2018, pp. 475–485.

[0257]

[65] M. Bōhme, V.-T. Pham, and A. Roychoudhury, “Coverage-based grey-box fuzzing as markov chain”, IEEE Transactions on Software Engineering, vol. 45, no. 5, pp. 489–506, 2017.

[0258]

[66] G. Zhang and X. Zhou, “'afl extended with test case prioritization techniques”, Int. J. Model. Optim, vol. 8, no. 1, pp. 41–45, 2018.

[0259]

[67] L. Moukahal and M. Zulkernine, “Security vulnerability metrics for connected vehicles,” in 2019 IEEE 19th International Conference on Software Quality, Reliability, and Security Guidelines (QRS-C), IEEE, 2019, pp. 17–23.

[0260]

[68] A.Zeller, R.Gopinath, M. G. Fraser and C. Holler, “The fuzzingbook,” CISPA Helmholtz Center for Information Security, 2019;

[0261]

[69] P. Chen and H. Chen, “Angora: Efficient fuzzing by principled search,” in 2018 IEEE Special Topic on Security and Privacy (SP), IEEE, 2018, pp. 711–725.

[0262]

[70] V. Wüstholz and M. Christakis, "Learning inputs in greybox fuzzing", arXiv preprint arXiv:1807.07875, 2018;

[0263]

[71] V. Jain, S. Rawat, C. Giuffrida, and H. Bos, “Tiff: using input typeinference to improve fuzzing,” in Proceedings of the 34th Annual Conference on Computer Security Applications, 2018, pp. 505–517.

[0264]

[72] S. Rawat and L. Mounier, “Offset-aware mutation based fuzzing for buffer overflow vulnerabilities: Few preliminary results”, in 2011 IEEE 4th International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2011, pp. 531–533.

[0265]

[73] R.C. Bhushan and D.D. Yadav, “Number of test cases required in achieving statement, branch and path coverage using ‘gcov’: An analysis,” in 7th International Symposium on Computer Science and Engineering (WCSE 2017), Beijing, China, 2017, pp. 176–180.

[0266] It should be understood that the described methods have been shown as individual steps performed in a specific order. However, those skilled in the art will appreciate that these steps can be combined or performed in a different order while still achieving the desired result.

[0267] It should be understood that a variety of different information processing systems can be used to implement embodiments of the present invention. In particular, although the figures and discussions thereof provide exemplary computing systems and methods, these are presented merely to provide a useful reference in discussing various aspects of the present invention. Embodiments of the present invention can be executed on any suitable data processing device, such as a personal computer, a laptop, a server computer, etc. Of course, for the purposes of discussion, the description of the systems and methods has been simplified, and they are merely one of the many different types of systems and methods that can be used for embodiments of the present invention. It should be understood that the boundaries between logic blocks are merely illustrative, and alternative embodiments can merge logic blocks or elements, or can impose alternative decompositions of functions on various logic blocks or elements.

[0268] It should be understood that the functions mentioned above can be implemented as one or more corresponding modules as hardware and / or software. For example, the functions mentioned above can be implemented as one or more software components for execution by the processor of the system. Alternatively, the functions mentioned above can be implemented as hardware, such as one or more field programmable gate arrays (FPGAs), and / or one or more application specific integrated circuits (ASICs), and / or one or more digital signal processors (DSPs), and / or one or more graphics processing units (GPUs), and / or other hardware arrangements. Each of the method steps implemented in the flowcharts included in this article or as described above can be implemented by the corresponding corresponding module; a plurality of method steps implemented in the flowcharts included in this article or as described above can be implemented together by a single module.

[0269] It should be understood that, insofar as the embodiments of the present invention are realized by computer programs, one or more storage media and / or one or more transmission media storing or carrying computer programs form various aspects of the present invention. A computer program can have one or more program instructions or program codes, which, when executed by one or more processors (or one or more computers), implement embodiments of the present invention. As used herein, term "program" can be an instruction sequence designed for execution on a computer system, and can include subroutines, functions, programs, modules, object methods, object implementations, executable applications, applet, servlet, source code, object code, bytecodes, shared libraries, dynamic link libraries, and / or other instruction sequences designed for execution on a computer system. Storage media can be disks (such as hard disk drives or floppy disks), optical disks (such as CD-ROMs, DVD-ROMs or Blu-ray discs), or memory (such as ROMs, RAMs, EEPROMs, EPROMs, flash memories, or portable / removable memory devices), etc. Transmission media can be communication signals, data broadcasts, communication links between two or more computers, etc.

Claims

1. A method for fuzz testing a software system, wherein the software system includes a plurality of callable units and is arranged to receive input for processing by the software system, the method comprising: determining, for each callable unit in the plurality of callable units, a target number of times to test the callable unit based on one or more security vulnerability metrics; Initializing a plurality of sorted queues, each queue being configured to store one or more seeds, wherein the initialization comprises storing the one or more initial seeds in a corresponding queue in the plurality of sorted queues; Executes a series of tests, where each test consists of: Get the seed from the highest sorted non-empty queue; performing a mutation process on the obtained seeds to generate test seeds; providing a test seed as input to the software system for processing by the software system; and evaluating the software system's processing of test seeds to generate test results; wherein each queue in the ordered plurality of queues has an associated seed addition criterion, and wherein performing each test comprises: (a) adding a test seed to a highest-ranked queue in the ordered plurality of queues for which the test seed satisfies the seed addition criterion associated with that queue; or (b) discarding the test seed if the test seed does not satisfy the seed addition criterion associated with any queue in the ordered plurality of queues; The seed addition criteria are configured such that: if the software system's processing of a first test seed involves the execution of a callable unit of interest or an execution path close to the callable unit of interest and if the software system's processing of a second test seed does not involve the execution of a callable unit of interest or an execution path close to the callable unit of interest, the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added, wherein if the current number of tests that result in the execution of a callable unit is less than the target number of times the callable unit is to be tested, the callable unit is a callable unit of interest.

2. The method according to claim 1, wherein The seed addition criteria are configured such that: if the software system's processing of a first test seed involves an execution path close to a callable unit of interest but does not involve execution of the callable unit of interest and if the software system's processing of a second test seed involves execution of the callable unit of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added.

3. The method according to claim 1, wherein The seed addition criteria are configured such that: if the software system's processing of a first test seed involves an execution path close to a callable unit of interest but does not involve execution of the callable unit of interest and if the software system's processing of a second test seed involves execution of the callable unit of interest, then the queue to which the first test seed is added has a lower ranking than the queue to which the second test seed is added.

4. The method according to any one of claims 1 to 3, wherein: The seed addition criteria are configured such that: if processing of a first test seed by the software system involves execution of one or more first callable units of interest or is proximate to an execution path of the one or more first callable units of interest and if processing of a second test seed by the software system involves execution of one or more second callable units of interest or is proximate to an execution path of the one or more second callable units of interest, then the queue to which the first test seed is added has a higher ranking than the queue to which the second test seed is added if: (a) a remaining number of times to be tested of at least one of the one or more first callable units of interest is greater than a remaining number of times to be tested of each of the one or more second callable units of interest; or (b) The sum of the remaining times for each of the one or more first callable units of interest to be tested is greater than the sum of the remaining times for each of the one or more second callable units of interest to be tested.

5. The method according to any one of claims 1 to 3, wherein: The seed addition criterion for the first queue is that the processing of the test seed by the software system involves the execution of the callable unit of interest or is close to the execution path of the callable unit of interest.

6. The method according to any one of claims 1 to 3, wherein: The seed addition criterion for the second queue is that the processing of the test seed by the software system reaches a branch point in the software system that was not reached when executing the previous test.

7. The method according to claim 6, wherein: The seeding criterion for the first queue is that processing of the test seed by the software system involves execution of the callable unit of interest or an execution path close to the callable unit of interest, and wherein the first queue has a higher ranking than the second queue.

8. The method according to claim 7, wherein: The ordered plurality of queues is a set including a first queue and a second queue.

9. The method according to any one of claims 1 to 3, wherein: Obtaining a seed from a highest-ranked non-empty queue includes removing the seed from the highest-ranked non-empty queue.

10. The method of any one of claims 1 to 3, comprising determining for a test seed a corresponding reuse quantity indicating the number of future tests for which the seed may be used as the obtained seed.

11. The method according to claim 10, wherein: Determining the corresponding reuse quantity for the test seed includes: If processing of the test seed by the software system involves execution of the callable unit of interest, setting the reuse quantity to a first predetermined value; If the processing of the test seed by the software system does not involve execution of the callable unit of interest but involves an execution path proximate to the callable unit of interest, setting the reuse quantity to a second predetermined value; If the software system's processing of the test seed does not involve execution of the callable unit of interest or an execution path close to the callable unit of interest but reaches a branch point in the software system that was not reached when executing previous tests, the reuse quantity is set to a third predetermined value.

12. The method according to claim 11, wherein Either: (a) the first predetermined value is greater than the second predetermined value, and the second predetermined value is greater than the third predetermined value; or (b) The second predetermined value is greater than the first predetermined value, and the first predetermined value is greater than the third predetermined value.

13. The method according to claim 10, comprising, for each stored seed, storing a corresponding reuse quantity, and wherein, Obtaining a seed from a highest-ordered non-empty queue includes decrementing a reuse number corresponding to the seed and (a) retaining the seed in the highest-ordered non-empty queue if the reuse number corresponding to the seed is not zero, and (b) removing the seed from the highest-ordered non-empty queue if the reuse number corresponding to the seed is zero.

14. The method according to claim 10, wherein: Adding a test seed to a highest-ranked queue among the sorted plurality of queues in which the test seed satisfies seed addition criteria associated with the queues comprises adding the test seed to a highest-ranked queue among the sorted plurality of queues in which the test seed satisfies seed addition criteria associated with the queues a number of times equal to the reuse number, and wherein obtaining the seed from the highest-ranked non-empty queue comprises removing the seed from the highest-ranked non-empty queue.

15. The method according to any one of claims 1 to 3, wherein: The mutation process performed on the obtained seeds to generate test seeds includes: (a) if the obtained seed is the initial seed, setting the test seed to the obtained seed; and (b) mutating the obtained seeds to further form test seeds.

16. The method according to any one of claims 1 to 3, wherein: When the one or more security vulnerability metrics indicate a higher level of security vulnerability for the callable unit, a higher target number is generated for determining a target number of times to test the callable unit for each of the plurality of callable units.

17. The method according to any one of claims 1 to 3, wherein: Initializing the ordered plurality of queues includes storing each of the one or more initial seeds in a highest ordered queue.

18. The method according to any one of claims 1 to 3, wherein The test sequence is executed until a termination condition is met, where the termination condition includes one or more of the following: (a) Each queue in the sorted plurality of queues is empty; (b) a threshold number of tests have been performed; and (c) A threshold amount of time is spent executing the test sequence.

19. The method according to any one of claims 1 to 3, wherein: Processing of a test seed by a software system is considered to involve an execution path approaching a first callable unit if the first callable unit is reachable from a most distant callable unit in a call graph of the software system, where the most distant callable unit is a callable unit in an execution path where no other callable unit in the execution path is farther in the call graph from a root node in the call graph, and: (a) the number of callable units between the furthest callable unit and the first callable unit in the call graph is at most a predetermined threshold; or (b) the number of callable units between the farthest callable unit and the root node in the call graph is at least a predetermined threshold; or (c) the amount of code above the furthest callable unit in the call graph is at least a predetermined threshold; or (d) the amount of code below the furthest callable unit in the call graph is at most a predetermined threshold; or (e) The amount of code between the farthest callable unit and the first callable unit in the call graph is at most a predetermined threshold.

20. A method according to any one of claims 1 to 3, comprising providing output for fuzz testing based on results generated from the performed testing.

21. The method according to any one of claims 1 to 3, wherein The software system is the vehicle's software system.

22. The method according to any one of claims 1 to 3, wherein Each callable unit is a respective one of: a routine; a subroutine; a function; a procedure; a process; a class method; an interface; a component; or a subsystem of a larger system.

23. The method according to any one of claims 1 to 3, wherein The one or more security vulnerability metrics include one or more of the following: (a) a measure indicating the degree of security vulnerability and / or security criticality of a callable unit; (b) a measure representing the risk that a malicious message can be passed from one callable unit to another; (c) metrics based on the number and / or type of communication technologies used by the callable units; (d) Code complexity metrics based on callable units; (e) a metric based on the number of callable function input and output parameters that have varying values ​​and / or the extent to which callable function input and output parameters can have varying values; as well as (f) Metrics based on historical vulnerability data associated with callable units.

24. A system arranged to implement the method according to any one of claims 1 to 23.

25. Computer readable medium storing a computer program which, when executed by one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 23.

Images