Realistic test scenario variation and test coverage generation for testing of autonomous vehicles

Description

[0001] REALISTIC TEST SCENARIO VARIATION AND TEST COVERAGE GENERATION FOR TESTING OF AUTONOMOUS VEHICLES

[0002] The present disclosure relates to testing of autonomous vehicle systems. More particularly, the present disclosure relates to a system and a method for generating realistic test scenarios employable in testing of an autonomous vehicle. Furthermore, the present disclosure relates to a system and a method for optimizing test coverage resulting from the realistic test scenarios generated and verification and validation of the autonomous vehicle systems based on the realistic test scenarios.

[0003] With rapid progress being made in the field of autonomous driving technology, gaining trust and confidence of drivers and passengers of the autonomous vehicles and acceptance from the environment in which these vehicles are expected to be driven, are necessary factors to foster the development of autonomous driving technologies. To achieve such trust and acceptance, it is imperative that the advanced driver assist systems (ADAS) and / or the automated driving system (ADS), associated with the autonomous vehicles as well as semi- autonomous vehicles, are thoroughly tested with a wide range of test scenarios. For example, an autonomous vehicle (AV) is required to exhibit about 8.8 billion failure free miles of travel to prove safe operation as compared to a human driven vehicle. Testing of the AV, that is the ADS and ADAS, in real world is tremendously costly as well as unsafe. Therefore, conventional testing often relies on simulation as a safe, reliable, cheap and reproducible alternative to real world testing.

[0004] However, it is challenging to make simulation test cases accurately represent real world test scenarios and scale the number of test cases concisely to cover various possible variations of a given test scenario. Conventionally, for a given test scenario description, test specifications are determined using brute-force combinatorics which involves writing functional test cases for vehicle software testing using in range values, out of range invalidity, functionality response over the variable range, etc., as some of the thumb rules. With ADAS software being injected more and more into an AV, the corresponding requirement for verification and validation of the ADAS also grows. Moreover, the inputs to ADAS systems are typically very complex and have heavy data storage and computational requirements such as readings from high- dimensional sensors such as camera, lidar and radar, etc. This means a large number of test scenarios and test cases are required to be tested for certification of ADAS modules. Therefore, traditional approaches for test case development are not scalable and feasible. Moreover, it is difficult to gauge a precise number of scenario variations to be tested for a given scenario description and to confirm test coverage, that is, whether and which part of the scenario variations was the ADS / ADAS tested for.

[0005] Furthermore, determining right values of scenario parameters to be used in test cases and identifying edge test cases and scenario parameter values corresponding to these edge test cases is another challenge.

[0006] Accordingly, it is an object of the present disclosure to provide a system and a method for generating realistic test scenarios with an optimized test coverage.

[0007] The present disclosure achieves the aforementioned object by providing a test scenario management system, hereinafter referred to as TSMS, and a computer implemented method for generating realistic test scenarios and an optimized test coverage. As used herein, the term “realistic” refers to quantifiable statistical properties derivable from real world data associated with the testing of the AVs. The statistical properties may be derived, for example, by determining a set of statistical properties used for comparing two datasets, computing those properties using real world data collected and synthetic data generated using various mechanisms and comparing the two. Moreover, statistical tests may be used to compare distributions of parameters from real world data and synthetic data to determine whether synthetic data is realistic or not.

[0008] The TSMS comprises a non-transitory computer readable storage medium storing computer program instructions defined by the TSMS, and at least one processor communicatively coupled to the non-transitory computer readable storage medium, wherein the at least one processor is configured to execute the computer program instructions, thereby performing the computer implemented method for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV).

[0009] According to an embodiment, the TSMS is in communication with an autonomous system employable in testing of the AVs. The autonomous system refers to a driving automation system spanning across various levels of driving automation and includes, for example, an automated driving system (ADS) and / or an advanced driver assistance system (ADAS). According to an embodiment, the TSMS communicates with the autonomous system via a communication network. The communication network is, for example, a wired network, a wireless network, or a network formed from any combination thereof.

[0010] According to an embodiment, the TSMS disclosed herein is installable on and accessible by a user device, for example, a personal computing device, a workstation, a client device, a network enabled computing device, any other suitable computing equipment, and combinations of multiple pieces of computing equipment being used by a user (not shown).

[0011] According to another embodiment, the TSMS may also be installable on the autonomous system.

[0012] According to yet another embodiment, the TSMS is configurable as a web-based platform, for example, a website hosted on a server or a network of servers, or, is implemented in the cloud computing environment as a cloud computing-based platform implemented as a service. A user of the TSMS in this case accesses the TSMS via the communication network. The TSMS may have one or more users for example, an AV stack tester.

[0013] According to an aspect, the TSMS disclosed herein comprises a non-transitory computer readable storage medium and at least one processor communicatively coupled to the non-transitory computer readable storage medium. As used herein, “non-transitory computer readable storage medium” refers to all computer readable media, for example, non-volatile media, volatile media, and transmission media except for a transitory, propagating signal. The non-transitory computer readable storage medium is configured to store computer program instructions defined by the TSMS. The processor is configured to execute the defined computer program instructions for generating realistic test scenarios and an optimized test coverage associated with testing of an AV.

[0014] According to another aspect, disclosed herein is the computer implemented method for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV). As used herein, the autonomous vehicle (AV) refers to any vehicle that is capable of being driven at least partially autonomously.

[0015] The computer implemented method disclosed herein employs the test scenario management system (TSMS) for generating realistic test scenarios and an optimized test coverage. The computer implemented method generates plurality of test scenario variations based on vehicle data associated with a plurality of AVs. The vehicle data is stored in the autonomous system. The computer implemented method receives the vehicle data from the autonomous system. The vehicle data comprises data recorded by a plurality of image capturing devices and by sensors mounted on the autonomous vehicles (AVs) and / or infrastructure sensors deployed in surroundings in which the AVs are being driven. The vehicle data comprises, for example, a type of a vehicle such as a bus, a car, etc., driver behaviour such as driver rashness score, timestamped vehicle trajectories consisting of coordinates, velocity, yaw angle, bounding box information associated wiht the vehicle and / or its surroundings, etc., responses if any of pedestrian(s) in proximity of the vehicle, from a geography in which the vehicle is being driven including road geometry such as lanes, angles, section lengths, coordinates of the junctions, etc.

[0016] Upon receiving the vehicle data, the computer implemented method extracts one or more scenes from the vehicle data. The scenes comprise, for example, junction description, vehicle trajectory and pedestrian trajectory at a particular time including behavioural parameters of both the vehicle and the pedestrian at the given time.

[0017] The computer implemented method generates a relational data model of traffic based on the scene(s) and the vehicle data, in a particular geographical region. The relational data model describes data relations between the scene(s) and the vehicle data, for example, by providing various statistics and reports such as at a given time of the day the average rashness in the extracted scene appears to increase by a certain percentage. The computer implemented method stores the relational data model in a relational database of the TSMS.

[0018] The computer implemented method extracts one or more events from the relational data model based on vehicle trajectories of the vehicles from the scene(s). The events comprise, for example, lane change event where a vehicle changes from a first lane to a second lane at a certain speed with a certain turn angle, etc.

[0019] The computer implemented method generates a semantic data model based on the event(s). The semantic data model describes vehicle behaviours of the vehicles with respect to the vehicle trajecotries associated thereiwth and geographical locations associated therewith. The semantic data model comprises, for example, nodes representing the vehicle, the lane in which the vehicle is being driven, the junction at which the lane ends, etc. The nodes are interconnected with arrows describing relationships between the different nodes, for example, a vehicle turns at a junction, a vehicle travels in a lane, etc. Thus, a vehicle’s journey and trajectory through a geolocation is represented by modelling the vehicle as a node and mapping variables associated therewith as other nodes and the driving behavior including, for example, an average speed, a turning angle, a trajectory curvature, etc., therebetween as arrows describing the relationships. The computer implemented method stores the semantic data model in a semantic database of the TSMS.

[0020] The computer implemented method, generates joint multivariate probability distributions of scenario parameters of a test scenario, by analyzing the event(s) in the semantic data model, for sampling the parameter values to create large number of test scenario variations.

[0021] As used herein, “test scenario” refers to a predefined condition or sequence of the events designed to evaluate performance, safety, and responsiveness of an autonomous vehicle. The test scenarios are structured representations that simulate specific scenes that the autonomous vehicle might encounter in real-world driving conditions. A test scenario comprises scenario parameters comprising, for example, road type, traffic conditions, weather situations, pedestrian actions, and other dynamic events. The test scenarios facilitate measurement of how well an autonomous vehicle can detect, interpret, and respond to the scenes or conditions, ensuring safe and efficient operation of the vehicle.

[0022] The computer implemented method analyzes the events in the semantic data model offline by using, for example, statistical generative learning techniques to create the multivariate joint probability distributions of the scenario parameters. The computer implemented method stores the test scenario variations generated in a test scenario database.

[0023] The computer implemented method determines a number of plausible test scenario variations, that is, realistically possible test scenario variations, from the plurality of test scenario variations, for a given Operational Design Domain (ODD) and / or a given Object and Event Detection and Response (OEDR). The computer implemented method determines the number of plausible test scenario variations based on a number of statistically significant relevant test scenarios that may be derived using a statistical technique. Advantageously, the computer implemented method for a given ODD and / or the OEDR and the vehicle data from the semantic data model, corresponding to a particular geographical location for a test scenario, determines a number of test scenarios that the vehicle is to be tested for, to issue a micro-certificate for scenario description.

[0024] The computer implemented method determines the number of possible test scenarios for given scenario parameters from a test scenario description and nominal ranges of values of the scenario parameters from the test scenario description, by employing a combinatorial method. The combinatorial method uses mathematical principles from combinatorics and with information on the physical constraints on the parameter values and combinatorial co-occurrence of the scenario parameters to get the number of test cases possible, that is, possible test scenarios.

[0025] The computer implemented method using the statistically significant relevant test scenarios, identifies a number of plausible test scenarios from the possible test scenarios.

[0026] The computer implemented method uses the real-world data to compute the values of the scenario parameters and abstracts the computed values into conditional or joint multivariate probability distributions for scenario variables using generative statistical learning techniques. These techniques include, for example, white box techniques such as fitting variables to predefined probability distribution functions or black box techniques based on neural networks such as generative adversarial networks or variational autoecoders, etc. Possible scenario variables that can be captured from real-world using different sensors comprise, for example, dynamic elements such as vehicles and pedestrians and their timestamped position trajectories, behavioral parameterization based on traffic, culture, region and road geometry, type of vehicle, behavioral classification based on the kind of driving, etc. The computer implemented method categorizes these variables based on their data type such that each variable will have different methods of generating realistic samples and variations. For example, for a scenario variable „ Vehicle type“, having possible scenario parameter values „Car“, „Bus“, „Truck“, etc., a combinatorics approach might be implemented to variate the the scenario parameter values, that is, to generate possible combinations of variables. Based on all possible values for each scenario variable the number of plausible scenarios to be tested for would be a combinatorial multiplication of the possibe values of the scenario variables. The computer implemented method generates a test coverage using the number of plausible test scenario variations, a probability of occurrence of a test scenario, and a severity of the test scenario. As used herein, „test coverage'' refers to a fraction of the plausible test scenarios that are simulated and tested.

[0027] The computer implemented method determines the probability of occurrence of the test scenario from the plausible test scenario varaitions.

[0028] The computer implemented method discretizes scenario variables of a test scenario variation using discretization bins, that is, by determining an optimal size of the discretization bins employed for discretizing the scenario variables. The computer implemented method advantageously, determines an optimal size of the bins so that an optimal doscretization is achieved while keeping the test scenario samples as diverse as possible to ensure an optimal test coverage. The computer implemented method achieves the optimization of the bin sizes based on a test coverage and a variability metric associated with the respective bins.

[0029] For example, the computer implemented method discretizes a given continuous joint distribution function of scenario variables into a number of bins. Given the number of bins, the computer implemented method identifies the bin boundaries, integrates probability density functions within each of the bins, and calculates a single likelihood number for each bin. The computer implemented method then normalizes these likelihoods to get probability values in bins so that the discretized representation has the characteristics of a probability density function, namely all values between 0 and 1 and net integral equal to 1. The discretized distribution is stored in a dictionary or look-up table like data structure in a bin discretization database of the TSMS.

[0030] The computer implemented method calculates the test coverage for a given discretized bin distribution, over a testspace, including distribution coverage, domain coverage, adversarial coverage, etc.

[0031] The computer implemented method calculates the variability metric based on a number of the discretization bins, distribution variance, and between -bin variance.

[0032] The variability metric is a product of two inversely moving expressions. As the number of discretization bins increases the between-bin variance value comes closer to the distribution variance as their difference decreases and vice versa. Therefore, optimizing the variability metric reduces the repetition / redundancy in the samples of the scenario variables.

[0033] The computer implemented method optimizes the test coverage and the variability metric for determining optimal bin sizes of the discretization bins, by employing a multi -objective search algorithm having the test coverage and the variability metric as the objectives to be optimized. The computer implemented method maintains a record of population samples, that is, the bin sizes that have been tried and evaluated. When determining the new bin size to be tried, the computer implemented method considers, approximate improvement that the new bin sizes could bring given the record / dataset of already tried bin sizes vs coverage / variability and the diversity in the test population, that is, how different the new bin size if from the already tried one. The computer implemented method achieves this using standard simulation-based optimization approaches such as Genetic Algorithms, Simulated Annealing, Reinforcement Learning, Stochastic Adaptive Search, etc.

[0034] The computer implemented method then computes the new bin size which is in turn used for next iteration of discretizing the scenario variables into new bins. The computer implemented method performs these iterations until a predefined iteration termination criterion is met comprising, for example, when a fixed number of iterations is completed, when convergence in the bin size values is observed providing a tradeoff between test coverage and the variability metric, when stagnation is observed wherein the new bin sizes are not changing significantly over multiple iterations, when a threshold for the test coverage and the variability metric is met, and / or when a domain specific stopping criteria are met that is test coverage criteria defined in testing standards of AVs such as ISO 26262, etc.

[0035] Thus, the computer implemented method optimizes bin sizes for said discretization of the scenario variables for maximizing test coverage of real world possibilities while minimizing the repetition of test cases and / or while maximizing variability in test cases.

[0036] The computer implemented method determines a likelihood of occurrence of the test scenarios associated with each of the discretization bins, such that the number of samples to be drawn from each discretization bin for defining the test coverage would be a function of the likelihood, that is, the probability of occurrence associated with the respective bin.

[0037] The computer implemented method determines the severity of the test scenario from the plausible test scenario varaitions. As used herein, „severity“ refers to the unsafeness of the test scenario wherein the unsafeness is a result of not only geometry of a driveable geographical area but also a combination of configuration and / or geometry of driveable geographical area, local traffic rules, driving behavior of the local demography, etc. Morevoer, unsafeness is also a function of other environmental factors which increase / decrease the individual unsafeness contribution by aforementioned parameters.

[0038] Thus, advantageously, the computer implemented method determines the severity, that is, unsafeness contribution of a test scenario resulting from traffic rules, deviations of traffic rules, driver immaturity, and the geometry of the driveable geographical area itself, by using real-world and simulated traffic data and accident data to determine individual contribution to unsafeness of the aforementioned parameters.

[0039] The computer implemented method trains an unsafeness determination model to jointly predict unsafeness of a test scenario as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules.

[0040] The computer implemented method generates safety analysis data based on the real world traffic data and the simulated traffic data. The safety analysis data includes data associated with the driving behaviour and with the traffic rules.

[0041] In order to generate the safety analysis data associated with the driving behaviour, the computer implemented method selects a geolocation and a set of traffic rules at that geolocation for collecting data from real world, analyses the driving behaviors of recorded vehicle trajectories in terms of deviations from the traffic rules, simulates large amount of traffic data by varying the behavior of the vehicles while keeping the traffic rules and road geometry constant, and analyzes this real-world data and simulated data to quantify safety criticality and unsafeness of the test scenarios. Thus, the computer implemented method generates a massive and rich dataset of unsafeness as a function of varying driving behavior. The computer implemented method stores this safety analysis data pertaining to driver behaviour into a safety analysis database of the TSMS.

[0042] In order to generate the safety analysis data associated with the traffic rules, the computer implememted method selects test scenarios having the same geolocation and compliant driving behavior, but varies the traffic rules at that geolocation, that is, the computer implemented method simulates large amount of traffic data by varying the traffic rule parameters while keeping the driving behavior always compliant with respect to the input traffic rules and road geometry constant. For the real-world data in this part, the computer implemented method collects data from multiple different geolocations with similar driveable geographical area but different set of traffic rules and analyses the driving behaviors of recorded vehicle trajectories in terms of deviations from the traffic rules prescriptions. The computer implemented method filters out the compliant behaviors and generates a dataset of driving at nearly the same road geometries and varied traffic rules but fully compliant driving behaviors. The computer implemented method analyzes this real and simulated data to quantify the safety criticality and unsafeness of the test scenarios as a function of varying traffic rules. The computer implemented method stores this safety analysis data pertaining to varying traffic rules into the safety analysis database of the TSMS.

[0043] The computer implemented method uses the safety analysis data stored in the safety analysis database to train the unsafeness determination model to jointly predict unsafeness of a test scenario as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules. This trained model employs artificial intelligence and / or machine learning techniques to learn how to jointly predict unsafeness as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules. This trained model is deployable in generating safety-relevant sampling of test cases for testing and / or verification and validation of autonomous Driving Systems and / or generating inputs for improving / maturing traffic rules or infrastructure.

[0044] The computer implemented method trains a safety criticality determination model to predict safety criticality of a test scenario based on combinations of values of scenario parameters of the test scenario, for example, the values that form the test scenario.

[0045] In order to train the safety criticality determination model, the computer implemented method generates crash data based on historical crash data, near crash driving dataset and / or historical test records from simulation and / or real- world and stores the same into a crash database of the TSMS. The computer implemented method uses the crash database to train the safety criticality determination model for the aforementioned purpose of predicting the safety criticality of a test scenario given the values of parameter combinations forming the test scenario. This trained model employs artificial intelligence and / or machine learning techniques to learn how to predict the safety criticality of a test scenario given the values of parameter combinations forming the test scenario. The computer implemented method predicts an unsafeness score associated with a test scenario by employing the trained unsafeness determination model and the trained safety criticality determination model. This is achieved by using the general unsafeness contribution, for each value of a parameter of a test scenario or combinations of parameters of a test scenario, to weigh sampling probability of a value of parameter or combination of parameters.

[0046] Thus, the computer implemented method predicts the unsafeness score, that is, the severity associated with various values of parameters of a test scenario.

[0047] According to an embodiment, the computer implemented method can assign the safety criticality weightage to various ranges of parameter values of the test scenario and use that for sampling as opposed to sampling from continuous joint distributions only.

[0048] The computer implemented method, determines the test coverage based on the probability of occurrence of the test scenario and the severity of the test scenario such that the test coverage is a product of the probability of occurrence of the test scenario and the severity of the test scenario.

[0049] According to an embodiment, the computer implemented method may generate 2- dimensional plots of the probability of occurrence versus the severity to determine the test coverage.

[0050] According to another embodiment, the computer implemented method may prioritize the scenarios with high likelihood, that is, high probability of occurrence and high criticality, that is, high severity or unsafeness scores to determine the test coverage.

[0051] Moreover, the computer implemented method may, on a user request, generate one or more of a test coverage report associated with the ODD and / or the OEDR based on the real world data, a number of experiments to test to cover one test scenario domain, a number of possible test scenarios as a result of the combinatorial approach, a number of plausible scenarios as a result of the statistical approach, semantic driving data models, and realistic specifications of a test scenario based on real-world data. According to yet another aspect, disclosed herein is a computer program product comprising a non-transitory computer readable storage medium that stores one or more computer program codes comprising instructions executable by at least one processor for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV).

[0052] The above mentioned and other features of the invention will now be addressed with reference to the accompanying drawings of the present invention. The illustrated embodiments are intended to illustrate, but not limit the invention.

[0053] The present invention is further described hereinafter with reference to illustrated embodiments shown in the accompanying drawings, in which:

[0054] FIG 1 illustrates a system having a test scenario management system for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

[0055] FIG 2 is a block diagram illustrating an architecture of a computer system employed by the test scenario management system shown in FIG 1, for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

[0056] FIGS 3A-3F illustrate process flowcharts of a computer implemented method for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

[0057] Various embodiments are described with reference to the drawings, wherein like reference numerals are used to refer like elements throughout. In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide thorough understanding of one or more embodiments. It may be evident that such embodiments may be practiced without these specific details.

[0058] FIG 1 illustrates a system having a test scenario management system (TSMS) 102 for generating realistic test scenarios and an optimized test coverage associated with testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure. The TSMS 102 is in communication with an autonomous system 101 employable in testing of the AVs. The autonomous system 101 refers to a driving automation system spanning across various levels of driving automation and includes, for example, an automated driving system (ADS) and / or an advanced driver assistance system (ADAS).

[0059] The TSMS 102 communicates with the autonomous system 101 via a communication network 103. The communication network 103 is, for example, a wired network, a wireless network, or a network formed from any combination thereof.

[0060] The TSMS 102 disclosed herein is installable on and accessible by a user device, for example, a personal computing device, a workstation, a client device, a network enabled computing device, any other suitable computing equipment, and combinations of multiple pieces of computing equipment being used by a user (not shown). The TSMS 102 may also be installable on the autonomous system 101.

[0061] The TSMS 102 is configurable as a web-based platform, for example, a website hosted on a server or a network of servers, or, is implemented in the cloud computing environment as a cloud computing-based platform implemented as a service. A user of the TSMS 102 in this case accesses the TSMS 102 via the communication network 103. The TSMS 102 may have one or more users for example, an AV stack tester.

[0062] The TSMS 102 disclosed herein comprises a non-transitory computer readable storage medium and at least one processor communicatively coupled to the non- transitory computer readable storage medium. As used herein, “non-transitory computer readable storage medium” refers to all computer readable media, for example, non-volatile media, volatile media, and transmission media except for a transitory, propagating signal. The non-transitory computer readable storage medium is configured to store computer program instructions defined by the TSMS 102. The processor is configured to execute the defined computer program instructions for generating realistic test scenarios and an optimized test coverage associated with testing of an AV, as described in the detailed description of FIGS 3A-3F.

[0063] The TSMS 102 comprises a graphical user interface (GUI) 104, and a database 105 representing several databases 105A-105E of the TSMS 102. A user using the user device can access the TSMS 102 via the GUI 104. The GUI 104 is, for example, an onhne web interface, a web based downloadable application interface, etc. The database 105 comprises, for example, a relational database 105A, a semantic database 105B, a test scenario database 1050, a bin discretization database 105D, and a safety analysis database 105E.

[0064] FIG 2 is a block diagram illustrating an architecture of a computer system 200 employed by the test scenario management system (TSMS) 102 shown in FIG 1, for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

[0065] The TSMS 102 employs the architecture of the computer system 200. The computer system 200 is programmable using a high-level computer programming language. The computer system 200 may be implemented using programmed and purposeful hardware. The computer system 200 comprises a processor 201, a non-transitory computer readable storage medium such as a memory unit 202 for storing programs and data, an input / output (I / O) controller 203, a network interface 204, a data bus 205, a display unit 206, input devices 207, a fixed media drive 208 such as a hard drive, a removable media drive 209 for receiving removable media, output devices 210, etc.

[0066] The processor 201 refers to any one of microprocessors, central processing unit (CPU) devices, finite state machines, microcontrollers, digital signal processors, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., or any combination thereof, capable of executing computer programs or a series of commands, instructions, or state transitions. The processor 201 may also be implemented as a processor set comprising, for example, a general-purpose microprocessor and a math or graphics co-processor. The TSMS 102 disclosed herein is not limited to a computer system 200 employing a processor 201. The computer system 200 may also employ a controller or a microcontroller. The processor 201 executes the instructions defined by the TSMS 102.

[0067] The memory unit 202 is used for storing programs and applications of the TSMS 102. The memory unit 202 is, for example, a random-access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by the processor 201. The memory unit 202 also stores temporary variables and other intermediate information used during execution of the instructions by the processor 201. The computer system 200 further comprises a read only memory (ROM) or another type of static storage device that stores static information and instructions for the processor 201. The I / O controller 203 controls input actions and output actions performed by the TSMS 102.

[0068] The network interface 204 enables connection of the computer system 200 to the communication network 103. For example, the TSMS 102 connects to the communication network 103 via the network interface 204. In an embodiment, the network interface 204 is provided as an interface card also referred to as a line card. The network interface 204 comprises, for example, interfaces using serial protocols, interfaces using parallel protocols, and Ethernet communication interfaces, interfaces based on wireless communications technology such as satellite technology, radio frequency (RF) technology, near field communication, etc. The data bus 205 permits communications between, for example, the TSMS 102, the database 105, the GUI 104, etc.

[0069] The display unit 206, via the graphical user interface (GUI) 104, displays information such as the test scenario set Ts, the focused test scenario set Tsf, etc. The display unit 206, via the GUI 104, also displays information such as user interface elements including text fields, buttons, windows, etc., for allowing a user to provide his / her inputs, if any. The display unit 206 comprises, for example, a liquid crystal display, a plasma display, an organic light emitting diode (OLED) based display, etc. The input devices 207 are used for inputting data into the computer system 200. The input devices 207 are, for example, a keyboard such as an alphanumeric keyboard, a touch sensitive display device, and / or any device capable of sensing a tactile input.

[0070] Computer applications and programs are used for operating the computer system 200. The programs are loaded onto the fixed media drive 208 and into the memory unit 202 of the computer system 200 via the removable media drive 209. In an embodiment, the computer applications and programs may be loaded directly via the communication network 103. Computer applications and programs are executed by double clicking a related icon displayed on the display unit 206 using one of the input devices 207. The output devices 210 output the results of operations performed by the TSMS 102. For example, the TSMS 102 provides graphical representation of the various scores such as an unsafeness score associated with a test scenario using the output devices 210.

[0071] The processor 201 executes an operating system. The computer system 200 employs the operating system for performing multiple tasks. The operating system is responsible for management and coordination of activities and sharing of resources of the computer system 200. The operating system further manages security of the computer system 200, peripheral devices connected to the computer system 200, and network connections. The operating system employed on the computer system 200 recognizes, for example, inputs provided by the users using one of the input devices 207, the output display, files, and directories stored locally on the fixed media drive 208. The operating system on the computer system 200 executes different programs using the processor 201. The processor 201 and the operating system together define a computer platform for which application programs in high level programming languages are written.

[0072] The processor 201 of the computer system 200 employed by the TSMS 102 retrieves instructions defined by the TSMS 102 for performing respective functions disclosed in the detailed description of FIGS 3A-3F. A program counter determines the location of the instructions in the memory unit 202. The instructions fetched by the processor 201 from the memory unit 202 after being processed are decoded. The instructions are stored in an instruction register in the processor 201. After processing and decoding, the processor 201 executes the instructions, thereby performing one or more processes defined by those instructions.

[0073] At the time of execution, the instructions stored in the instruction register are examined to determine the operations to be performed. The processor 201 then performs the specified operations. The operations comprise arithmetic operations and logic operations. The operating system performs multiple routines for performing several tasks required to assign the input devices 207, the output devices 210, and memory for execution of the instructions defined by the TSMS 102. The tasks performed by the operating system comprise, for example, assigning memory as required by the TSMS 102, moving data between the memory unit 202 and disk units, and handling input / output operations. The operating system performs the tasks on request by the operations and after performing the tasks, the operating system transfers the execution control back to the processor 201. The processor 201 continues the execution to obtain one or more outputs. The outputs of the TSMS 102 are displayed to the user on the GUI 104.

[0074] For purposes of illustration, the detailed description refers to the TSMS 102 being run locally on the computer system 200, however the scope of the present invention is not limited to the TSMS 102 being run locally on the computer system 200 via the operating system and the processor 201, but may be extended to run remotely over the communication network (not shown) by employing a web browser and a remote server, a mobile phone, or other electronic devices. One or more portions of the computer system 200 may be distributed across one or more computer systems (not shown) coupled to the communication network.

[0075] Disclosed herein is also a computer program product comprising a non-transitory computer readable storage medium that stores computer program codes comprising instructions executable by at least one processor 201 for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), as disclosed in detailed description of FIGS 3A-3F.

[0076] A person having ordinary skills in the art would appreciate that the computer program product would comprise several computer program codes for executing instructions defined by the TSMS 102 for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), as disclosed in detailed description of FIGS 3A-3F.

[0077] FIGS 3A-3F illustrate process flowcharts of a computer implemented method 300 for generating realistic test scenarios and an optimized test coverage employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure. As used herein, the autonomous vehicle (AV) refers to any vehicle that is capable of being driven at least partially autonomously.

[0078] The computer implemented method 300 disclosed herein employs the test scenario management system, hereinafter referred to as TSMS 102, shown in FIG 1 for generating realistic test scenarios and an optimized test coverage.

[0079] At step 301, the computer implemented method 300 generates plurality of test scenario variations based on vehicle data associated with a plurality of AVs. The vehicle data is stored in the autonomous system 101 shown in FIG 1.

[0080] At step 301 A, the computer implemented method 300 establishes communication between the autonomous system 101 and the TSMS 102, for example, via the communication network 103. Upon successfully establishing the communication therebetween, at step 30 IB, the computer implemented method 300 receives the vehicle data. The vehicle data comprises data recorded by a plurality of image capturing devices and by sensors mounted on the autonomous vehicles (AVs) and / or infrastructure sensors deployed in surroundings in which the AVs are being driven. The vehicle data comprises, for example, a type of a vehicle such as a bus, a car, etc., driver behaviour such as driver rashness score, timestamped vehicle trajectories consisting of coordinates, velocity, yaw angle, bounding box information associated wiht the vehicle and / or its surroundings, etc., responses if any of pedestrian(s) in proximity of the vehicle, from a geography in which the vehicle is being driven including road geometry such as lanes, angles, section lengths, coordinates of the junctions, etc.

[0081] Upon receiving the vehicle data, the computer implemented method 300, at step 301C, extracts one or more scenes from the vehicle data stored in the autonomous system 101. The scenes comprise, for example, junction description, vehicle trajectory and pedestrian trajectory at a particular time including behavioural parameters of both the vehicle and the pedestrian at the given time.

[0082] At step 30 ID, the computer implemented method 300 generates a relational data model of traffic based on the scene(s) and the vehicle data, in a particular geographical region. The relational data model describes data relations between the scene(s) and the vehicle data, for example, by providing various statistics and reports such as at a given time of the day the average rashness in the extracted scene appears to increase by a certain percentage. The computer implemented method 300 stores the relational data model in a relational database 105A of the TSMS 102.

[0083] At step 30 IE, the computer implemented method 300 extracts one or more events from the relational data model based on vehicle trajectories of the vehicles from the scene(s). The events comprise, for example, lane change event where a vehicle changes from a first lane to a second lane at a certain speed with a certain turn angle, etc.

[0084] At step 30 IF, the computer implemented method generates a semantic data model based on the event(s). The semantic data model describes vehicle behaviours of the vehicles with respect to the vehicle trajecotries associated thereiwth and geographical locations associated therewith. The semantic data model comprises, for example, nodes representing the vehicle, the lane in which the vehicle is being driven, the junction at which the lane ends, etc. The nodes are interconnected with arrows describing relationships between the different nodes, for example, a vehicle turns at a junction, a vehicle travels in a lane, etc. Thus, a vehicle’s journey and trajectory through a geolocation is represented by modelling the vehicle as a node and mapping variables associated therewith as other nodes and the driving behavior including, for example, an average speed, a turning angle, a trajectory curvature, etc., therebetween as arrows describing the relationships. The computer implemented method 300 stores the semantic data model in a semantic database 105B of the TSMS 102.

[0085] At step 301G, the computer implemented method 300, generates joint multivariate probability distributions of scenario parameters of a test scenario, by analyzing the event(s) in the semantic data model, for sampling the parameter values to create large number of test scenario variations.

[0086] As used herein, “test scenario” refers to a predefined condition or sequence of the events designed to evaluate performance, safety, and responsiveness of an autonomous vehicle. The test scenarios are structured representations that simulate specific scenes that the autonomous vehicle might encounter in real-world driving conditions. A test scenario comprises scenario parameters comprising, for example, road type, traffic conditions, weather situations, pedestrian actions, and other dynamic events. The test scenarios facilitate measurement of how well an autonomous vehicle can detect, interpret, and respond to the scenes or conditions, ensuring safe and efficient operation of the vehicle.

[0087] The computer implemented method 300 analyzes the events in the semantic data model offline by using, for example, statistical generative learning techniques to create the multivariate joint probability distributions of the scenario parameters. The computer implemented method 300 stores the test scenario variations generated in a test scenario database 105C.

[0088] At step 302, the computer implemented method 300 determines a number of plausible test scenario variations, that is, realistically possible test scenario variations, from the plurality of test scenario variations, for a given Operational Design Domain (ODD) or a given Object and Event Detection and Response (OEDR). The computer implemented method determines the number of plausible test scenario variations based on a number of statistically significant relevant test scenarios that may be derived using a statistical technique.

[0089] Advantageously, the computer implemented method 300 for a given ODD and / or the OEDR and the vehicle data from the semantic data model, corresponding to a particular geographical location for a test scenario, determines a number of test scenarios that the vehicle is to be tested for, to issue a micro-certificate for scenario description. At step 302A, the computer implemented method 300 determines the number of possible test scenarios for given scenario parameters from a test scenario description and nominal ranges of values of the scenario parameters from the test scenario description, by employing a combinatorial method. The combinatorial method uses mathematical principles from combinatorics and with information on the physical constraints on the parameter values and combinatorial co-occurrence of the scenario parameters to get the number of test cases possible, that is, possible test scenarios.

[0090] At step 302B, the computer implemented method 300 using the statistically significant relevant test scenarios, identifies a number of plausible test scenarios from the possible test scenarios.

[0091] The computer implemented method 300 uses the real-world data to compute the values of the scenario parameters and abstracts the computed values into conditional or joint multivariate probability distributions for scenario variables using generative statistical learning techniques. These techniques include, for example, white box techniques such as fitting variables to predefined probability distribution functions or black box techniques based on neural networks such as generative adversarial networks or variational autoecoders, etc.

[0092] For example, possible scenario variables that can be captured from real- world using different sensors are:

[0093] • Dynamic elements such as vehicles and pedestrians and their timestamped position trajectories

[0094] • behavioral parameterization based on traffic, culture, region and road geometry, type of vehicle

[0095] • behavioral classification based on the kind of driving rash / compliant / mixed

[0096] • traffic jam - macroscopic influence - mass behavior

[0097] • weather conditions

[0098] • static road elements

[0099] • vehicle condition also affect behavior - vehicle aging model

[0100] • time of the day and time of the year

[0101] • legal infractions if any can be enlisted with

[0102] These variables can then be categorized based on their data type and each will have different methods of generating realistic samples and variations as shown in the Table 1 below.

[0103] Table 1 Based on all possible values for each scenario variable the number of plausible scenarios to be tested for would be a combinatorial multiplication of the possibe values of the scenario variables. At step 303, the computer implemented method 300 generates a test coverage using the number of plausible test scenario variations, a probability of occurrence of a test scenario, and a severity of the test scenario. As used herein, „test coverage'' refers to a fraction of the plausible test scenarios that are simulated and tested.

[0104] At step 304, the computer implemented method determines the probability of occurrence of the test scenario from the plausible test scenario varaitions.

[0105] At step 304A, the computer implemented method discretizes scenario variables of a test scenario variation using discretization bins, that is, by determining an optimal size of the discretization bins employed for discretizing the scenario variables. The computer implemented method advantageously, determines an optimal size of the bins so that an optimal doscretization is achieved while keeping the test scenario samples as diverse as possible to ensure an optimal test coverage. The computer implemented method achieves the optimization of the bin sizes based on a test coverage and a variability metric associated with the respective bins.

[0106] For example, the computer implemented method discretizes a given continuous joint distribution function of scenario variables into a number of bins. Given the number of bins, the computer implemented method identifies the bin boundaries, integrates probability density functions within each of the bins, and calculates a single likelihood number for each bin. The computer implemented method then normalizes these likelihoods to get probability values in bins so that the discretized representation has the characteristics of a probability density function, namely all values between 0 and 1 and net integral equal to 1. The discretized distribution is stored in a dictionary or look-up table like data structure in a bin discretization database 105D of the TSMS 102.

[0107] At Step 304B, the computer implemented method calculates the test coverage for a given discretized bin distribution, over a testspace, including distribution coverage, domain coverage, adversarial coverage, etc.

[0108] At step 304C, the computer implemented method calculates the variability metric based on a number of the discretization bins, distribution variance, and between-bin variance. The variability metric thus calculated may be represented by the equation givne below:

[0109] VM = (BBV - DV) * Bn Wherein VM is the variabihty metric, BBV is the between-bin variance, DV is the distribution variance and Bn is the number of the discretization bins.

[0110] Thus, the variabihty metric VM is a product of two inversely moving expressions. As the number of discretization bins Bn increases the between-bin variance BBV value comes closer to the distribution variance DV as their difference decreases and vice versa. Therefore, optimizing the variability metric VM reduces the repetition / redundancy in the samples of the scenario variables.

[0111] At step 304D, the computer implemented method optimizes the test coverage and the variabihty metric for determining optimal bin sizes of the discretization bins, by employing a multi-objective search algorithm having the test coverage and the variability metric as the objectives to be optimized. The computer implemented method maintains a record of population samples, that is, the bin sizes that have been tried and evaluated. When determining the new bin size to be tried, the computer implemented method considers, approximate improvement that the new bin sizes could bring given the record / dataset of already tried bin sizes vs coverage / variability and the diversity in the test population, that is, how different the new bin size if from the already tried one. The computer implemented method achieves this using standard simulation-based optimization approaches such as Genetic Algorithms, Simulated Annealing, Reinforcement Learning, Stochastic Adaptive Search, etc. The computer implemented method then computes the new bin size which is in turn used for next iteration of discretizing the scenario variables into new bins. The computer implemented method performs these iterations until a predefined iteration termination criterion is met comprising, for example, when a fixed number of iterations is completed, when convergence in the bin size values is observed providing a tradeoff between test coverage and the variability metric, when stagnation is observed wherein the new bin sizes are not changing significantly over multiple iterations, when a threshold for the test coverage and the variability metric is met, and / or when a domain specific stopping criteria are met that is test coverage criteria defined in testing standards of AVs such as ISO 26262, etc.

[0112] Thus, at steps 304A-304D, the computer implemented method optimizes bin sizes for said discretization of the scenario variables for maximizing test coverage of real world possibilities while minimizing the repetition of test cases and / or while maximizing variability in test cases. At step 304E, the computer implemented method determines a likelihood of occurrence of the test scenarios associated with each of the discretization bins, such that the number of samples to be drawn from each discretization bin for defining the test coverage would be a function of the likelihood, that is, the probability of occurrence associated with the respective bin.

[0113] For example, consider a test scenario of a target vehicle turning at an intersection. The main scenario variables of concern for this test scenario comprise turning speed, turning angle, turning trajectory, the map geometry, etc. To generate significant number of test scenario variations to cover this test scenario, the possible ranges and values for each scenario variable are identified followed by an empirical likelihood of each value or a range of values for each scenario variable. Now for test scenario distributions that are skewed, for example, for turning angle, less number of samples need to be drawn from each bin as these would cover the possibilities to sufficient extent, whereas for distributions that are not skewed for example for turning speed, a large number of samples need to be drawn from each bin as the scenario distribution would be much flatter in some of the ranges of the scenario variables. The sampling from these distributions is performed till the extent that the number of samples become statistically significant. This can be achieved by computing the statistical parameters on the samples such as mean, variance, skewness and kurtosis, and comparing the values of these statistical parameters with those computed for the full scenario distribution. If these statistical parameters become same or almost same, then it can be assumed that enough samples are drawn from each bin to represent the underlying scenario distribution fairly.

[0114] At step 305, the computer implemented method determines the severity of the test scenario from the plausible test scenario varaitions.

[0115] As used herein, „severity“ refers to the unsafeness of the test scenario wherein the unsafeness is a result of not only geometry of a driveable geographical area but also a combination of configuration and / or geometry of driveable geographical area, local traffic rules, driving behavior of the local demography, etc. Morevoer, unsafeness is also a function of other environmental factors which increase / decrease the individual unsafeness contribution by aforementioned parameters.

[0116] Thus, advantageously, the computer implemented method determines the severity, that is, unsafeness contribution of a test scenario resulting from traffic rules, deviations of traffic rules, driver immaturity, and the geometry of the driveable geographical area itself, by using real-world and simulated traffic data and accident data to determine individual contribution to unsafeness of the aforementioned parameters.

[0117] At step 305A, the computer implemented method trains an unsafeness determination model to jointly predict unsafeness of a test scenario as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules.

[0118] At step 305B, the computer implemented method generates safety analysis data based on the real world traffic data and the simulated traffic data. The safety analysis data includes data associated with the driving behaviour and with the traffic rules.

[0119] In order to generate the safety analysis data associated with the driving behaviour, the computer implemented method selects a geolocation and a set of traffic rules at that geolocation for collecting data from real world, analyses the driving behaviors of recorded vehicle trajectories in terms of deviations from the traffic rules, simulates large amount of traffic data by varying the behavior of the vehicles while keeping the traffic rules and road geometry constant, and analyzes this reabworl data and simulated data to quantify safety criticality and unsafeness of the test scenarios. Thus, the computer implemented method generates a massive and rich dataset of unsafeness as a function of varying driving behavior. The computer implemented method stores this safety analysis data pertaining to driver behaviour into a safety analysis database 105E of the TSMS 102.

[0120] In order to generate the safety analysis data associated with the traffic rules, the computer implememted method selects test scenarios having the same geolocation and compliant driving behavior, but varies the traffic rules at that geolocation, that is, the computer implemented method simulates large amount of traffic data by varying the traffic rule parameters while keeping the driving behavior always compliant with respect to the input traffic rules and road geometry constant. For the real-world data in this part, the computer implemented method collects data from multiple different geolocations with similar driveable geographical area but different set of traffic rules and analyses the driving behaviors of recorded vehicle trajectories in terms of deviations from the traffic rules prescriptions. The computer implemented method filters out the compliant behaviors and generates a dataset of driving at nearly the same road geometries and varied traffic rules but fully compliant driving behaviors. The computer implemented method analyzes this real and simulated data to quantify the safety criticality and unsafeness of the test scenarios as a function of varying traffic rules. The computer implemented method stores this safety analysis data pertaining to varying traffic rules into the safety analysis database 105E of the TSMS 102.

[0121] The computer implemented method uses the safety analysis data stored in the safety analysis database 105E to train the unsafeness determination model to jointly predict unsafeness of a test scenario as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules, as mentioned in step 305A. This trained model employs artificial intelligence and / or machine learning techniques to learn how to jointly predict unsafeness as a function of traffic rule parameters and driving behavior in terms of deviations from those traffic rules. This trained model is deployable in generating safety-relevant sampling of test cases for testing and / or verification and validation of autonomous Driving Systems and / or generating inputs for improving / maturing traffic rules or infrastructure.

[0122] At step 305C, the computer implemented method trains a safety criticality determination model to predict safety criticality of a test scenario based on combinations of values of scenario parameters of the test scenario, for example, the values that form the test scenario.

[0123] In order to train the safety criticality determination model, the computer implemented method, at step 305D, generates crash data based on historical crash data, near crash driving dataset and / or historical test records from simulation and / or real- world and stores the same into a crash database 105F of the TSMS 102. The computer implemented method uses the crash database 105F to train the safety criticality determination model, as per step 305C, for the aforementioned purpose of predicting the safety criticality of a test scenario given the values of parameter combinations forming the test scenario. This trained model employs artificial intelligence and / or machine learning techniques to learn how to predict the safety criticality of a test scenario given the values of parameter combinations forming the test scenario.

[0124] At step 305E, the computer implemented method predicts an unsafeness score associated with a test scenario by employing the trained unsafeness determination model and the trained safety criticality determination model. This is achieved by using the general unsafeness contribution, for each value of a parameter of a test scenario or combinations of parameters of a test scenario, to weigh sampling probability of a value of parameter or combination of parameters. For example, consider sampling values 1 and 2 of a single parameter of a test scenario, from the real world distributions present in the safety analysis database 105E, provide an unsafeness probability of 0.4 for value_l and 0.6 for value_2. Similarly, sampling the values 1 and 2 of the single parameter of the test scenario, from the historical crash / near-crash datasets present in the crash database 105F provides relative safety criticality of 0.7 for value_l and 0.3 for value_2. Then using the safety criticalities as weighting factors for the unsafeness probabilities, a net unsafeness score, that is, unsafeness-weighted likelihood values of 04*0.7 = 0.28 for value_l and 0.6*0.3 = 0.18 for value_2 can be obtained. Thus, the sample value_l has a higher chance of being sampled and consequently higher number of representative samples in the test cases generated / used for testing the ADS.

[0125] Thus, at step 305, the computer implemented method predicts the unsafeness score, that is, the severity associated with various values of parameters of a test scenario.

[0126] According to an embodiment, the computer implemented method can assign the safety criticality weightage to various ranges of parameter values of the test scenario and use that for sampling as opposed to sampling from continuous joint distributions only.

[0127] The computer implemented method, at step 306, determines the test coverage based on the probability of occurrence of the test scenario, determined at step 304, and the severity of the test scenario, determined at step 305. The test coverage is a product of the probability of occurrence of the test scenario and the severity of the test scenario. The computer implemented method may generate 2 -dimensional plots of the probability of occurrence versus the severity to determine the test coverage. Alternatively, the computer implemented method may prioritize the scenarios with high likelihood, that is, high probability of occurrence and high criticality, that is, high severity or unsafeness scores to determine the test coverage.

[0128] Moreover, the computer implemented method may, on a user request, generate one or more of a test coverage report associated with the ODD and / or the OEDR based on the real world data, a number of experiments to test to cover one test scenario domain, a number of possible test scenarios as a result of the combinatorial approach, a number of plausible scenarios as a result of the statistical approach, semantic driving data models, and realistic specifications of a test scenario based on real-world data. Where databases are described such as the relational database 105A, the semantic database 105B, the test scenario database 105C, the bin discretization database 105D, the safety analysis database 105E, etc., it will be understood by one of ordinary skill in the art that (i) alternative database structures to those described may be readily employed, and (ii) other memory structures besides databases may be readily employed. Any illustrations or descriptions of any sample databases disclosed herein are illustrative arrangements for stored representations of information. Any number of other arrangements may be employed besides those suggested by tables illustrated in the drawings or elsewhere. Similarly, any illustrated entries of the databases represent exemplary information only! one of ordinary skill in the art will understand that the number and content of the entries can be different from those disclosed herein. Further, despite any depiction of the databases as tables, other formats including relational databases, object-based models, and / or distributed databases may be used to store and manipulate the data types disclosed herein. Likewise, object methods or behaviors of a database can be used to implement various processes such as those disclosed herein. In addition, the databases may, in a known manner, be stored locally or remotely from a device that accesses data in such a database. In embodiments where there are multiple databases in the system, the databases may be integrated to communicate with each other for enabling simultaneous updates of data linked across the databases, when there are any updates to the data in one of the databases.

[0129] The present disclosure can be configured to work in a network environment comprising one or more computers that are in communication with one or more devices via a network. The computers may communicate with the devices directly or indirectly, via a wired medium or a wireless medium such as the Internet, a local area network (LAN), a wide area network (WAN) or the Ethernet, a token ring, or via any appropriate communications mediums or combination of communications mediums. Each of the devices comprises processors, some examples of which are disclosed above, that are adapted to communicate with the computers. In an embodiment, each of the computers is equipped with a network communication device, for example, a network interface card, a modem, or other network connection device suitable for connecting to a network. Each of the computers and the devices executes an operating system, some examples of which are disclosed above. While the operating system may differ depending on the type of computer, the operating system will continue to provide the appropriate communications protocols to establish communication links with the network. Any number and type of machines may be in communication with the computers. The present disclosure is not limited to a particular computer system platform, processor, operating system, or network. One or more aspects of the present disclosure may be distributed among one or more computer systems, for example, servers configured to provide one or more services to one or more client computers, or to perform a complete task in a distributed system. For example, one or more aspects of the present disclosure may be performed on a client-server system that comprises components distributed among one or more server systems that perform multiple functions according to various embodiments. These components comprise, for example, executable, intermediate, or interpreted code, which communicate over a network using a communication protocol. The present disclosure is not limited to be executable on any particular system or group of systems, and is not limited to any particular distributed architecture, network, or communication protocol.

[0130] While the present invention has been described in detail with reference to certain embodiments, it should be appreciated that the present invention is not limited to those embodiments. In view of the present disclosure, many modifications and variations would be present themselves, to those skilled in the art without departing from the scope of the various embodiments of the present invention, as described herein. The scope of the present invention is, therefore, indicated by the following claims rather than by the foregoing description. All changes, modifications, and variations coming within the meaning and range of equivalency of the claims are to be considered within their scope.

Claims

CLAIMS1. A computer implemented method (300) for generating realistic test scenarios and an optimized test coverage associated with testing of an autonomous vehicle, the computer implemented method characterized by: generating (301) a plurality of test scenario variations based on vehicle data associated with a plurality of autonomous vehicles; determining (302) a number of plausible test scenario variations from the plurality of test scenario variations for one or more of a given operational design domain (ODD) and a given object and event detection and response (OEDR); and generating (303) a test coverage using the number of plausible test scenario variations, a probability of occurrence of a test scenario, and a severity of the test scenario.

2. The method according to claim 1, wherein generating the test scenario variations based on vehicle data associated with a plurality of autonomous vehicles comprises: extracting one or more scenes from the vehicle data stored in an autonomous sytem; generating a relational data model of traffic based on the one or more scenes and the vehicle data, wherein the relational data model describes data relations between the one or more scenes and the vehicle data; extracting one or more events from the relational data model based on vehicle trajectories of the vehicles from the one or more scenes; generating semantic data model based on the one or more events, wherein the semantic driving data model describes vehicle beahviors of the vehicles with respect to the vehicle trajecotries associated thereiwth and geographical locations associated therewith; and generating joint multivariate probability distributions of scenario parameters of a test scenario by analyzing the one or more events in the semantic data model.

3. The method according to claim 1, wherein determining the number of plausible test scenario variations from the plurality of test scenario variations comprises: determining the number of possible test scenarios for given scenario parameters, and nominal ranges of values of the scenario parameters, from a test scenario description, by employing a combinatorial method; and identifying a number of realistically possible test scenarios from the number of possible test scenarios using a statistical method.

4. The method according to claim 1, wherein generating the test coverage using the number of plausible test scenario variations, a probability of occurrence of a test scenario and a severity of the test scenario, comprises determining the probability of occurrence of the test scenario from the plausible test scenario variations; determining the severity of the test scenario from the plausible test scenario variations; and determining the test coverage based on the probability of occurrence of the test scenario and the severity of the test scenario.

5. The method according to claim 4, wherein determining the probability of occurrence of the test scenario from the plausible test scenario variations comprises ■ discretizing scenario variables of a test scenario variation using discretization bins; calculating test coverage for a given discretized bin distribution; calculating a variability metric based on a number of the discretization bins, distribution variance, and between-bin variance; optimizing the test coverage and the variability metric for determining optimal bin sizes of the discretization bins; and determining a likelihood of occurrence of the test scenarios associated with each of the discretization bins.

6. The method according to claim 4, wherein determining the severity of the test scenario from the plausible test scenario variations comprises: training an unsafeness determination model to jointly predict unsafeness of a test scenario as a function of traffic rule parameters and driving behavior in terms of deviations from traffic rules; generating safety analysis data based on real world traffic data and simulated traffic data; training a safety criticality determination model to predict safety criticality of a test scenario based on combinations of values of scenario parameters of the test scenario; and predicting an unsafeness score associated with a test scenario by employing the trained unsafeness determination model and the trained safety criticality determination model.

7. The method according to claim 4, wherein the test coverage is a product of the probability of occurrence of the test scenario and the severity of the test scenario.

8. A test scenario management system (102) for generating realistic test scenarios and an optimized test coverage associated with testing of an autonomous vehicle, characterized by: - a non-transitory computer readable storage medium storing computer program instructions defined by the test scenario management system (102); at least one processor communicatively coupled to the non-transitory computer readable storage medium, wherein the at least one processor is configured to execute the computer program instructions, thereby performing the method according to the claims 1 to 7.

9. A computer-program product having machine-readable instructions stored therein, which when executed by one or more processors, cause the processors to perform the method according to the claims 1 to 7.

Images