Software driven system validated based on real world scenarios

Abstract

The present invention provides an apparatus (105) and a method for validating a software driven system based on real-world scenarios in a computer aided engineering environment (100). The method includes obtaining, by a processing unit (120), a plurality of test scenarios corresponding to a test of the software driven system. Further, at least one real-world scenario associated with the software driven system is generated based on a set of variable parameters. Further, one or more test scenarios suitable for testing the software driven system based on the at least one real-world scenario are identified from the plurality of test scenarios using a trained machine learning model. The identified test scenarios are applied on a model of the software driven system in a simulated environment to evaluate a behavior of the software driven system. Based on a result of the evaluation, a behavior of the software driven system in the real-world scenario is validated.

Description

[0001] The present invention relates to the field of verifying software driven systems, and more specifically, to an apparatus and method for verifying software driven systems based on real world scenarios.

[0002] Manufacturing companies usually perform a large number of tests on their products before putting them into the market. Such tests are performed to verify the specifications of the product against predefined standards or regulatory compliance. The specifications can correspond to functional or non-functional aspects of the product. If the product passes all the prescribed verification tests, the product is considered ready for launch. Otherwise, the manufacturing company can redesign the product in order to ensure that the specifications meet the predefined standards or regulatory compliance.

[0003] However, testing of complex systems such as components of autonomous systems running in different environments poses several challenges. Testing of complex systems can be cumbersome as the behavior of such autonomous systems is non-deterministic and is based on the context of the run. Further, as the complex systems are not always under the control or supervision of humans, it is important to ensure safe behavior of such systems through verification and validation (V&V) methods.

[0004] Complex systems are usually tested through simulation of real world scenarios. Such testing usually involves many experimental combinations. Each experimental combination can correspond to different settings of variable parameters associated with the environment of the complex system. However, some experimental combinations can be infeasible or impossible in real world scenarios. Further, as the simulation involved in the testing is computationally expensive, infeasible scenarios add to the computational complexity.

[0005] In view of the foregoing, there exists a need for a method and apparatus for verifying software driven systems based on feasible real world scenarios.

[0006] It is therefore an object of the present invention to provide an apparatus and method for verifying software driven systems based on feasible real world scenarios.

[0007] The object of the present invention is achieved by the method disclosed herein for validating a software-driven system based on real-world scenarios in a computer-aided engineering environment. Non-limiting examples of software-driven systems include machines, embedded systems, vehicles, autonomous systems, automated guided vehicles, and other complex systems operating in different real-world conditions. In embodiments, the software-driven system can have non-deterministic behavior in the real world. For example, the behavior of such a software-driven system can be governed by stimuli from the real world perceived by sensors associated with the software-driven system. Such stimuli are collectively referred to as sensory inputs hereinafter. In another example, the software-driven system can also operate based on inputs received from a human operator or another system through a communication interface.

[0008] The method includes obtaining a plurality of test scenarios corresponding to a test of the software-driven system. In embodiments of the present invention, the plurality of test scenarios are obtained from a source in the form of variable parameters. The source can be an input device, a user device, or a database. The variable parameters can include at least one of attributes and process parameters associated with the software-driven system. The attributes of the software-driven system include physical or functional characteristics resulting from the design of the software-driven system. The process parameters correspond to environmental factors associated with the software-driven system. Based on the set of variable parameters, a design space is generated. Here, the term "design space" refers to the multi-dimensional combinations and interactions of the variable parameters. The design space contains all possible combinations of the variable parameters. Each combination of the variable parameters in the design space corresponds to a test scenario or a design point. The design space can include both feasible test scenarios and infeasible test scenarios. The feasible test scenarios include test scenarios suitable for testing the software-driven system. The infeasible test scenarios include test scenarios unsuitable for testing the software-driven system. For example, the infeasible test scenarios can include scenarios that do not occur in the real world.

[0009] The method further includes generating at least one real-world scenario associated with the software-driven system based on the set of variable parameters. In embodiments, generating the at least one real-world scenario includes determining one or more constraints associated with the variable parameters. Further, the one or more constraints are applied to the design space so as to generate a pruned design space corresponding to the at least one real-world scenario.

[0010] Advantageously, the present invention facilitates pruning of the design space for testing the software-driven system by applying one or more constraints to the design space. Thus, unrealistic scenarios that do not occur in the real world are excluded from the test scenarios.

[0011] The method further includes identifying, using the trained machine learning model, one or more test scenarios from the plurality of test scenarios that are suitable for testing the software-driven system based on the at least one real-world scenario. The identified one or more test scenarios are feasible test scenarios to be used for testing the software-driven system. The trained machine learning model can be any trained function that mimics cognitive functions associated with human thinking. In particular, the trained function is capable of adapting to new environments and detecting and inferring patterns through training based on a training data set. Generally, parameters of the trained function are adapted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, feature learning can be used. In particular, parameters of the trained function are capable of being adapted iteratively through several training steps. The trained machine learning model samples the pruned design space to generate feasible test scenarios for testing the software-driven system.

[0012] Advantageously, the present invention facilitates generating any number of test scenarios suitable for testing a software-driven system through iterative sampling of a design space while completely eliminating unrealistic test scenarios.

[0013] The method further includes generating, in the computer-aided engineering environment, a simulation environment representing the at least one real-world scenario in which the software-driven system is to be tested. The simulation environment includes simulated agents corresponding to agents in the real-world scenario and is generated using simulation tools in the computer-aided engineering environment. The term "agent" as used herein can indicate an object in a physical environment associated with the software-driven system. For example, if the software-driven system is an autonomous vehicle, the agents can include buildings, roads, trees, pedestrians, other vehicles, and the like. The simulated agents can be, for example, physics-based models associated with the agents used to simulate sensory inputs to the software-driven system. The simulated agents can be static or dynamic in nature. In embodiments, generating the simulation environment includes configuring the simulated agents to represent real-world conditions in which the software-driven system is expected to operate.

[0014] The method further includes evaluating behavior of the software-driven system by applying the identified test scenarios to a model of the software-driven system in the simulation environment. In embodiments, the method further includes generating, in the computer-aided engineering environment, a model of the software-driven system. The model can be, for example, an analytical model of the software-driven system in a machine executable form. In embodiments, evaluating behavior of the software-driven system includes generating simulation instances based on the identified one or more test scenarios. The term "simulation instance" as used herein can refer to a simulation thread that is independent of all other threads during execution, which represents a state of the simulation environment. Furthermore, the simulation results are analyzed to determine behavior of the software-driven system in the one or more test scenarios.

[0015] Advantageously, the present application facilitates testing of a software driven system in a simulated environment that simulates real world conditions. Thus, the behavior of the software driven system in the simulated environment corresponds to the actual behavior of the software driven system in real world conditions.

[0016] The method further comprises validating the behavior of the software driven system in real world scenarios based on the results of the evaluation. In an embodiment, validating the behavior of the software driven system comprises determining whether the behavior of the software driven system in real world scenarios meets expected criteria.

[0017] Advantageously, the present application facilitates validating the software driven system using only test scenarios that correspond to real world scenarios. Thus, contrary to the prior art, the present application does not require manual effort to eliminate unrealistic test scenarios from a plurality of test scenarios.

[0018] The method further comprises generating a notification on the graphical user interface indicating the validation result. The notification can indicate whether the validation of the software driven system was successful. If the validation was not successful, the notification can further indicate the test scenario during which the behavior of the software driven system did not meet the expected criteria.

[0019] Advantageously, the present application facilitates faster and more efficient validation of the software driven system as compared to the prior art due to less human intervention in the identification of feasible test scenarios.

[0020] In an embodiment, an apparatus for validating a software driven system based on real world scenarios is disclosed. The apparatus comprises a processor and a memory unit communicatively coupled to the one or more processing units. The memory unit comprises a test module stored in the form of machine readable instructions executable by the one or more processing units. The test module is configured to, upon execution, perform the method steps according to the method described above. The execution of the test module can also be performed using a co-processor such as a graphics processing unit (GPU), a field programmable gate array (FPGA), or a neural processing / computing engine. Further, the memory unit can also comprise a database.

[0021] In one embodiment, the apparatus is a cloud computing system having a cloud computing based platform configured to provide cloud services for validation of software driven systems based on real world scenarios. As used herein, “cloud computing” refers to a processing environment comprising configurable computing physical and logical resources, such as networks, servers, storage, applications, services, etc., and data distributed over a network, such as the Internet. The cloud computing platform can be implemented as a service for managing medical imaging devices. In other words, the cloud computing system provides on-demand network access to a shared pool of configurable computing physical and logical resources. The network is, for example, a wired network, a wireless network, a communication network, or a network formed by any combination of these networks.

[0022] The application relates in one aspect to a computer readable medium on which program code sections of a computer program are stored, which program code sections are loadable into and / or executable in a data processing system, such that when the program code sections are executed in the data processing system, the data processing system carries out a method according to an aspect of the application.

[0023] The implementation of the application by means of a computer program product and / or a computer readable medium has the advantage that already existing test systems can be easily adopted by means of a software update in order to work as proposed by the application.

[0024] The computer program product can be, for example, a computer program or comprise another element in addition to the computer program. This other element can be: hardware, such as a memory device on which the computer program is stored, a hardware key for using the computer program, etc., and / or software, such as a document or a software key for using the computer program.

[0025] The above-mentioned properties, features and advantages of the application as well as the way in which they are achieved will become more apparent and comprehensible (clear) from the following description of embodiments of the application in conjunction with the corresponding drawings. The described embodiments are intended to illustrate but not to limit the application.

[0026] In the following, the application will be further described with reference to the illustrative embodiments shown in the drawings, in which:

[0027] Figure 1A A block diagram of an environment for validating an apparatus of a software-driven system based on a feasible real-world scenario according to embodiments of the application is shown;

[0028] Figure 1B A block diagram of an environment for validating an apparatus of a software-driven system based on a feasible real-world scenario according to embodiments of the application is shown;

[0029] Figure 2 is a flowchart depicting a method 200 for validating a software-driven system based on a feasible real-world scenario according to embodiments of the application;

[0030] Figure 3 is a flowchart depicting an exemplary method of obtaining a plurality of test scenarios for testing an autonomous vehicle according to embodiments of the application;

[0031] Figure 4 is a flowchart depicting an exemplary method of generating a real-world scenario associated with an autonomous vehicle based on a set of variable parameters according to embodiments of the application;

[0032] Figure 5A graphical user interface is shown that illustrates a two-dimensional view of a design space to which constraints associated with testing of autonomous vehicles in the real world have been applied, in accordance with embodiments of the present disclosure;

[0033] Figure 6 is a flowchart depicting an exemplary method for identifying test scenarios suitable for testing autonomous vehicles based on real-world scenarios using a trained machine learning model, in accordance with embodiments of the present invention;

[0034] Figure 7 is a flowchart depicting an exemplary method for generating test scenarios suitable for testing autonomous vehicles based on optimization criteria, in accordance with embodiments of the present invention;

[0035] Figure 8 is a flowchart depicting an exemplary method for determining test scenarios suitable for testing autonomous vehicles, in accordance with another embodiment of the present invention;

[0036] Figure 9 is a flowchart depicting an exemplary method for identifying test scenarios suitable for testing autonomous vehicles, in accordance with yet another embodiment of the present invention; and

[0037] Figure 10 is a flowchart depicting an exemplary method for evaluating behavior of autonomous vehicles, in accordance with embodiments of the present invention.

[0038] In the following, embodiments for carrying out the present invention are described in detail. Various embodiments are described with reference to the accompanying drawings, wherein like reference numerals are used throughout to refer to like elements. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It can be apparent, however, that such embodiments, can be practiced without

[0039] Figure 1A A block diagram of an environment 100 for validating a device 105 of a software-driven system (not shown) based on real-world scenarios, in accordance with embodiments of the present invention, is shown. More specifically, the environment 100 is a cloud-based system that facilitates computer-aided engineering. The device 105 is communicatively coupled to one or more user devices 110 over a network 115. The user devices 110 can be, for example, human-machine interfaces that enable test personnel to interact with the device 105.

[0040] As Figure 1BAs shown in the figure, the apparatus 105 comprises a processing unit 120, a memory 125, a storage unit 130, a communication unit 135, a network interface 140, an input unit 145, an output unit 150, a standard interface or bus 155. The apparatus 105 can be a (personal) computer, a workstation, a virtual machine running on host hardware, a microcontroller, or an integrated circuit. As an alternative, the apparatus 105 can be a real computer group or a virtual computer group (the technical term for a real computer group is "cluster", the technical term for a virtual computer group is "cloud"). The term "processing unit" as used herein means any kind of computing circuitry, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing microprocessor, a reduced instruction set computing microprocessor, a very long instruction word microprocessor, an explicitly parallel instruction computing microprocessor, a graphics processing unit, a digital signal processor, or any other kind of processing circuitry.

[0041] The processing unit 120 can further contain embedded controllers, such as general or programmable logic devices or arrays, application specific integrated circuits, single-chip computers, etc. Generally, the processing unit 120 can comprise hardware elements and software elements. The processing unit 120 can be configured for multi-threading, i.e. the processing unit 120 can host different computing processes at the same time, execute parallel active and passive computing processes or switch between active and passive computing processes.

[0042] The memory 125 can comprise one or more of volatile memory and non-volatile memory. The memory 125 can be coupled for communication with the processing unit 120. The processing unit 120 can execute instructions and / or code stored in the memory 125. A variety of computer-readable storage media can be stored in the memory 125 and accessed by the processing unit 120. The memory 125 can include any suitable element for storing data and machine-readable instructions, such as read-only memory, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, hard disk drives, removable media drives for processing compact disks, digital video disks, magnetic disks, magnetic cassettes, memory cards, and so on. The memory 125 comprises a test module 160, which can be stored in the memory 125 in the form of machine-readable instructions and executable by the processing unit 120. When executed by the processing unit 120, these machine-readable instructions cause the processing unit 120 to perform validation of a software-driven system based on real-world scenarios. The test module 160 comprises a pre-processing module 165, a scenario identification module 170, a test case generation module 175, an evaluation module 180, a validation module 185, and a report generation module 190.

[0043] The pre-processing module 165 is configured to obtain a plurality of test scenarios corresponding to a test of the software driven system. In the present embodiment, the plurality of test scenarios are obtained from the user device 110 in the form of variable parameter definitions. Further, the pre-processing module 165 generates a design space based on the variable parameter definitions.

[0044] The scenario identification module 170 is configured to generate at least one real-world scenario associated with the software driven system based on the set of variable parameters. The test case generation module 175 is configured to identify one or more test scenarios suitable for testing the software driven system based on the at least one real-world scenario from the plurality of test scenarios using the trained machine learning model.

[0045] The evaluation module 180 generates a simulated environment representative of the real-world scenario in which the software driven system is to be tested. The evaluation module 180 is further configured to evaluate the behavior of the software driven system by applying the identified test scenarios on the model of the software driven system in the simulated environment. The verification module 185 is configured to verify the behavior of the software driven system in the real-world scenario based on the result of the evaluation. The report generation module 190 is configured to generate a notification indicative of the verification result on the graphical user interface.

[0046] The storage unit 130 includes a non-volatile memory storing a database 195 including default values of the constraints and variable parameter definitions. The input unit 145 can include input means such as a keypad, a touch-sensitive display, a camera, etc. capable of receiving input signals. The bus 155 serves as an interconnect between the processing unit 120, the memory 125, the storage unit 130, and the network interface 140.

[0047] The communication unit 135 enables the apparatus 105 to communicate with one or more user devices 110. The communication unit 135 can support different standard communication protocols such as Transmission Control Protocol / Internet Protocol (TCP / IP), Profinet, Profibus, Bluetooth, and Internet Protocol version (IPv). The network interface 140 enables the apparatus 105 to communicate with the user device 110 through the network 115.

[0048] The apparatus 105 according to embodiments of the present application includes an operating system that employs a graphical user interface. The operating system allows multiple display windows to be presented simultaneously in the graphical user interface, each display window providing an interface to a different application or to a different instance of the same application. A cursor in the graphical user interface can be manipulated by the tester through a pointing device. The position of the cursor can be changed and / or an event such as clicking a mouse button can be generated to prompt a desired response.

[0049] One of various commercial operating systems, such as Microsoft Windows®, can be employed, with a suitable modification, if necessary.TM version. The operating system is modified or created in accordance with the described application.

[0050] Those of ordinary skill in the art will realize, Figure 1A and 1B The hardware depicted in the preceding figures can vary depending on the implementation. For example, other peripheral devices such as disk drives, network adapters, graphics adapters, etc. can be used in addition to or in place of the hardware depicted. The depicted example is not meant to imply architectural limitations. For example, the architecture of the present application can also be used with other computer systems or platforms.

[0051] The present application is not limited to a particular computer system platform, processing unit 120, operating system, or network. One or more aspects of the present application can be distributed among one or more computer systems (e.g., a server configured to provide one or more services to one or more client computers or perform a complete task in a distributed system). For example, one or more aspects of the present application can be executed on a client-server system that includes components distributed among one or more server systems performing various functions according to various embodiments. These components include, for example, executable code, intermediate code, or interpreted code that communicate over a network using a communication protocol. The present application is not limited to execution on any particular system or group of systems, and is not limited to any particular distributed architecture, network, or communication protocol.

[0052] Figure 2 is a flowchart depicting a method 200 for validating a software driven system based on real world scenarios in accordance with embodiments of the present application.

[0053] At step 205, a plurality of test scenarios corresponding to a test of the software driven system are obtained from a source. In a preferred embodiment, a test personnel enters variable parameter definitions associated with each variable required to perform the test through a graphical user interface of the user device 110, which are further transmitted to the apparatus 110. For example, the variable parameter definitions include an identifier and a range of values associated with each variable parameter. The variable parameters are selected such that the test of the software driven system based on various combinations of the variable parameters provides quality assurance of the software driven system. In an example, the test can be performed based on the variable parameters to check if the operation of the software driven system meets regulatory compliance. In another example, the test can be performed to determine if the software driven system meets a predetermined design specification.

[0054] At step 210, at least one real-world scenario associated with the software-driven system is generated based on the set of variable parameters. In a preferred embodiment, the at least one real-world scenario is generated by first determining one or more constraints associated with the variable parameters. Further, the one or more constraints are applied to the design space in order to generate a pruned design space corresponding to the at least one real-world scenario. In an example, the constraints can be explicitly defined by a tester. In another example, the constraints are specified graphically by the tester. In another example, in case the constraints are not known or provided, a machine learning model can be used to determine the constraints. More specifically, random test scenarios are generated by sampling the design space. Further, the tester can accept or reject each test scenario based on the feasibility of the test scenario. Further, the machine learning model can determine the constraints based on the acceptance or rejection of the scenarios, for example, based on a Convex-Hull approach.

[0055] At step 215, one or more test scenarios suitable for testing the software-driven system based on the at least one real-world scenario are identified from the plurality of test scenarios using a trained machine learning model. The trained machine learning model can include a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained function can be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. In particular, the neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Further, the neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network. In an embodiment, the samples are generated from the pruned design space using the trained machine learning model. The samples can be generated by feeding an input corresponding to the pruned design space to the trained machine learning model. In response to the input, the trained machine learning model generates the samples from the pruned design space. Further, the one or more test scenarios are identified based on the generated samples. For example, each sample can be a tuple including values of the variable parameters to be used in a particular test scenario.

[0056] In another embodiment, an optimal sample is generated from the pruned design space based on at least one optimization criterion. In an embodiment, an optimization function based on evolutionary algorithms can be used to generate the optimal sample. More specifically, random samples are generated from the pruned design space using a random sampling algorithm. Further, a cost criterion associated with the random samples is computed. Further, based on the values of the cost criterion computed for each random sample, a predefined number of random samples are selected. The selected random samples are further provided as input to the optimization function to generate the optimal sample. Further, the one or more test scenarios are identified based on the generated optimal sample. Each optimal sample can be a tuple including values of the variable parameters to be used in a particular test scenario.

[0057] At step 220, a simulated environment representing a real-world scenario in which the software-driven system is to be tested is generated. In an embodiment, generating the simulated environment includes configuring the simulated environment based on the test type corresponding to one or more test scenarios. For example, in case of an autonomous robot, the simulated environment can be configured to represent an outdoor condition or an indoor condition. In case of an autonomous vehicle, the simulated environment can be configured to represent a bumpy road, a smooth road, a traffic jam, and so on. More specifically, simulated agents in the simulated environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate.

[0058] At step 225, the behavior of the software-driven system is evaluated by applying the identified test scenarios on a model of the software-driven system in the simulated environment. For example, the model is an analytical model of the software-driven system in a machine executable form. The model can be generated based on, for example, a digital twin of the software-driven system. In an embodiment, the behavior of the software-driven system is evaluated by first generating simulation instances based on the identified one or more test scenarios. Each simulation instance is generated by updating the simulated environment based on each of the identified test scenarios to simulate inputs to the software-driven system. The simulation inputs can include sensory inputs, human inputs, and inputs from other systems. Further, each simulation instance is executed based on the model of the software-driven system to generate simulation results. The simulation instances can be executed using simulation tools such as stochastic simulation, deterministic simulation, dynamic simulation, continuous simulation, discrete simulation, local simulation, distributed simulation, and so on. During the execution of the simulation instance, the simulation inputs are applied to the model of the software-driven system. Based on the simulation inputs, the model generates responses. The simulation results indicate the responses of the model to the simulation inputs. Further, the simulation results are analyzed to determine the behavior of the software-driven system in the one or more test scenarios. The responses of the model indicate the behavior of the software-driven system in the real-world scenario. Thus, the responses of the model are analyzed to determine the behavior of the software-driven system in the real-world scenario.

[0059] At step 230, the behavior of the software-driven system in the real-world scenario is validated based on the results of the evaluation. In an embodiment, validating the behavior of the software-driven system includes determining whether the behavior of the software-driven system in the real-world scenario satisfies an expected standard. The expected standard can be associated with a regulatory compliance, a safety standard, or a predetermined specification, and so on. If the behavior of the software-driven system satisfies the expected standard, then the result of the validation is successful. Otherwise, the result of the validation is unsuccessful.

[0060] Each of the above steps is described in more detail using Figures 3 to 10

[0061] Figure 3 ​is a flow diagram depicting an exemplary method 300 of obtaining a plurality of test scenarios for testing an autonomous vehicle, according to an embodiment of the present application. The method includes steps 305 and 310.

[0062] At step 305, variable parameter definitions corresponding to variable parameters required for testing the autonomous vehicle are obtained from the user device 110. For example, the test can be associated with the braking function of the autonomous vehicle. Variable parameters affecting the braking function can include visibility in front of the autonomous system, weather, and coefficient of friction between the tires of the autonomous vehicle and the road. The variable parameter "weather" can take five values: '0' (for dry or sunny weather), '1' (for foggy or cloudy weather), 2 (for light rain), 3 (for heavy rain), and 4 (for snow). The variable parameter "coefficient of friction" can take five values: 0.9, 0.55, 0.4, 0.3, and 0.2. The variable parameter visibility can take five values: 1 (for normal visibility), 0.8 (for lower visibility). It must be understood that the possible range of values can be specified as discrete levels, continuous ranges. The variable parameter definitions can also be provided as probability distributions, for example, as shown below:

[0063] F1D1 ~ uniform(8, 30)

[0064] F1D2 ~ uniform(5, 30)

[0065] F2D1 ~ uniform(8, 30)

[0066] F2D2 ~ uniform(5, 30)

[0067] In addition to the variable parameter definitions, the number of test scenarios required for testing the autonomous vehicle can also be defined by the tester. For example, the number of test scenarios required can be defined as 125. In another example, the number of test scenarios can be derived automatically based on the variable parameter definitions, based on pre-defined rules.

[0068] At step 310, a design space corresponding to the variable parameters is generated. In the present example, the design space is associated with the variable parameters "weather", "visibility", and "coefficient of friction". The design can include both feasible test scenarios and infeasible test scenarios.

[0069] Figure 4 is a flow diagram depicting an exemplary method 400 of generating real-world scenarios associated with an autonomous vehicle based on a set of variable parameters, according to an embodiment of the present application. The method includes steps 405 and 410.

[0070] At step 405, one or more constraints associated with the testing of the autonomous vehicle are obtained from the source. For example, the test scenario may be provided by a tester via user equipment 110. In another embodiment, the constraints may be predefined. Constraints can be explicitly defined using a programming language on the GUI of user equipment 110, such as as shown below:

[0071] If (weather == 1):

[0072] Coefficient of friction > 0.5 & visibility < 0.2

[0073] If (weather == 0):

[0074] Coefficient of friction > 0.5 & visibility > 0.8

[0075] If (weather == 4):

[0076] Coefficient of friction <= 0.2 & Visibility = 0.5

[0077] If (weather == 3):

[0078] 0.2 <= coefficient of friction < 0.4 & visibility = 0.6

[0079] In another example, if there are four variable parameters F1D1, F1D2, F2D1, and F2D2, the constraint can have the following form:

[0080] F1D1+F1D2<32,

[0081] F1D1+F1D2<23

[0082] In another implementation, constraints are provided graphically. For example, a tester can provide constraints by drawing boundaries in a two-dimensional view of the design space on the graphical user interface of user device 110. In another example, the tester may not provide any constraints. Figure 5 As shown, a two-dimensional view of the design space is displayed to the tester on the user device's GUI 500. Furthermore, a set of random samples in the design space can be highlighted, such as by dots. Additionally, the tester is provided with the option to select feasible test scenarios and reject infeasible test scenarios, for example, by selecting an "Accept" or "Reject" option next to each dot. Therefore, the tester can accept or reject random samples. In the GUI 500, checked samples indicate feasible test scenarios, while crossed-out samples indicate infeasible test scenarios. Based on the accepted samples, the device 105 can automatically determine constraints, such as using... Figure 9 The explanation given.

[0083] At step 410, a correlation between the variable parameters is determined. If there is a correlation among the variable parameters, method 415 is executed. If there is no correlation among the variables, a space filling algorithm is used to generate test scenarios from the design space corresponding to the variable parameters as shown in step 420. The space filling algorithm can include any sampling algorithm that can perform sampling in the design space. Each sample generated by the space filling algorithm corresponds to a test scenario. Depending on the type of space filling algorithm, the generated samples can include random samples or optimal samples. The space filling algorithm can generate optimal samples based on known coverage metrics such as inter-sample force, max-min, min-max, or entropy criteria. The coverage metrics ensure that the samples generated from the design space are well spaced apart.

[0084] Figure 6 is a flowchart depicting an exemplary method 600 of identifying test scenarios suitable for testing an autonomous vehicle based on at least one real-world scenario using a trained machine learning model, in accordance with an embodiment of the present application. More specifically, the present embodiment relates to the identification of one or more test scenarios when constraints are explicitly provided by a test personnel. Here, the design space is sampled using a machine learning algorithm to generate N number of required samples. For example, the test personnel can specify N as '125'.

[0085] At step 605, k-means clustering is applied to the design space to generate k clusters. Further, kxN random samples are generated from the k clusters.

[0086] At step 610, constraints are applied to the design space to determine infeasible design space. The infeasible design space corresponds to design points or test scenarios that are not suitable for testing the autonomous vehicle. The remaining design points in the design space correspond to test scenarios that are suitable for testing the autonomous vehicle. At step 615, the kxN random samples within the feasible design space are clustered using K-means clustering to form N clusters. Further, the centroids of the N clusters are computed.

[0087] At step 620, each centroid of the N clusters is checked to determine if any centroid falls in the infeasible design space. If not, step 625 is executed. Otherwise, step 605 is repeated until a predefined termination criterion is satisfied.

[0088] At step 625, the centroids of the N clusters are generated as samples or feasible test scenarios for testing the autonomous vehicle. The feasible test scenarios generated here form a first set of test scenarios.

[0089] At step 630, it is determined whether the first set of test scenarios is suitable for testing the autonomous vehicle. In an embodiment, the suitability of the first set of test scenarios is based on the number of feasible test scenarios present within the first set of test scenarios. If the number of feasible test scenarios is less than the required number of test scenarios, the method 700 is executed to generate more feasible test scenarios. Otherwise, the method 1000 is executed.

[0090] Figure 7 is a flowchart depicting an exemplary method 700 of generating test scenarios suitable for testing an autonomous vehicle based on an optimization criterion, in accordance with an embodiment of the present application. At step 705, Np random samples are generated from the design space using Latin Hypercube Design (LHD) of variables. At step 710, a cost criterion associated with each of the Np samples is computed. In the present embodiment, the cost criterion is based on the intra-sample repulsion corresponding to each sample. More specifically, the cost criterion is the sum of the intra-sample repulsion of each sample. The intra-sample repulsion is a measure of entropy, i.e., randomness, associated with the sample. In another embodiment, the cost criterion is the intra-sample distance, i.e., the distance between the sample and another sample.

[0091] At step 715, a predefined number of samples are selected from the Np samples based on a predefined criterion corresponding to the cost criterion. For example, the predefined number of samples can be N p / 2, and the predefined criterion can be such that the selected N p / 2 samples have a low value of the sum of intra-sample repulsion compared to any of the remaining N p / 2 samples. At step 720, the selected samples are optimized using an optimization function. In a preferred embodiment, the optimization function performs the optimization using an evolutionary algorithm, for example, but not limited to, a genetic algorithm. More specifically, the selected samples are used as initial values for performing the optimization. The genetic algorithm can perform a constrained optimization of the selected samples based on one or more constraints associated with the variable parameters.

[0092] At step 725, the optimization function generates a new set of optimal samples from the selected samples. At step 730, at least one termination criterion associated with the optimization is checked. The termination criterion can be, for example, checking whether the number of optimal samples generated is greater than or equal to the required number of samples. If the termination criterion is satisfied, the test scenarios corresponding to the optimal samples are determined at step 735. Otherwise, steps 710 to 730 are repeated until the termination criterion is satisfied.

[0093] Figure 8 is a flowchart depicting an exemplary method 800 for determining test scenarios suitable for testing an autonomous vehicle, in accordance with another embodiment of the present application. In the present embodiment, the constraints are defined graphically.

[0094] At step 805, a graphical representation of the design space is provided to the test personnel on a graphical user interface.

[0095] At step 810, the test personnel graphically divides the design space into feasible and infeasible test scenarios. For example, the test personnel can create a sketch of a feasible region in the design space. The feasible region includes feasible test scenarios suitable for testing the autonomous vehicle. Alternatively, the test personnel can define an infeasible region that includes infeasible test scenarios unsuitable for testing the autonomous vehicle.

[0096] At step 815, a parametric sketch of the feasible region in the design space is generated using a machine learning algorithm. The parametric sketch of the feasible region is further used to generate a pruned design space. Further, the method 600 is used to generate feasible test scenarios from the pruned design space.

[0097] Figure 9 is a flowchart depicting an exemplary method 900 for identifying test scenarios suitable for testing an autonomous vehicle, according to yet another embodiment of the present application. In this embodiment, the test personnel does not provide constraints.

[0098] At step 905, random test scenarios are generated by means of random sampling of the design space using a space-filling algorithm as previously explained.

[0099] At step 910, the test personnel is provided with an option to accept feasible test scenarios and reject infeasible test scenarios.

[0100] At step 915, a pattern corresponding to the test personnel accepting feasible test scenarios is determined using a machine learning algorithm. The pattern can be determined using a convex hull method. The convex hull method can be, for example, the Graham algorithm that creates a boundary of the feasible design space path based on the feasible test scenarios accepted by the test personnel. Further, the method 600 is executed.

[0101] In identifying test scenarios, the evaluation module 180 generates a simulated environment representing a real-world scenario in which the autonomous vehicle is to be tested. In the present example of the autonomous vehicle, a simulated environment representing driving conditions in the real-world is generated based on each test scenario identified. When the autonomous vehicle is being tested to verify the braking function, simulated agents corresponding to weather, visibility, and coefficient of friction are configured. The simulated agents are configured based on physics-based models.

[0102] Figure 10 is a flowchart depicting an exemplary method 1000 for evaluating autonomous vehicle behavior, according to an embodiment of the present application.

[0103] At step 1005, a simulation instance for testing the autonomous vehicle based on the identified test scenarios is generated. The simulation instance is generated by updating the simulation environment based on the identified test scenarios to simulate the inputs to the autonomous vehicle. More specifically, the simulation inputs represent sensory inputs corresponding to weather, visibility, and friction coefficient. Thus, a simulation instance corresponding to each of the identified test scenarios is generated.

[0104] At step 1010, the simulation instance is executed based on the model of the autonomous vehicle to generate simulation results. The model of the autonomous vehicle is configured to represent the operational characteristics of the autonomous vehicle. During the execution of the simulation instance, the simulation inputs are provided as inputs to the model in order to generate responses from the model. The model can correspond to the braking functionality associated with the autonomous vehicle.

[0105] At step 1015, the responses from the model are further analyzed to determine the behavior of the autonomous vehicle in the identified test scenarios. In the present example, the behavior of the autonomous system when brakes are applied in a given test scenario can correspond to the stopping distance associated with the autonomous vehicle. The stopping distance is the sum of the reaction distance and the braking distance. For example, the output of the model can indicate both the reaction distance and the braking distance in microseconds.

[0106] The verification module 185 verifies the behavior of the autonomous vehicle based on the expected criteria. For example, as per the expected criteria, the stopping distance should not exceed 50 meters when the autonomous vehicle is operated at 50 kmpH. Similarly, the criteria can be defined based on different conditions. If the stopping distance exceeds the value defined by the criteria in any of the test scenarios, the verification of the autonomous vehicle is unsuccessful. Otherwise, the autonomous vehicle is successfully verified. The report generation module 190 can also generate a report indicating the verification results. In one implementation, the report can indicate the test scenarios used and the assessment results corresponding to each of the test scenarios. The generated report can further be transmitted to the user device 110 for display on the graphical user interface associated with the user device 110.

[0107] In comparison to the prior art, the present invention provides accurate and more reliable verification results by eliminating all unrealistic test scenarios from the design space. Further, the present invention also eliminates the manual effort required to identify feasible test scenarios, thereby enabling faster verification of software driven systems.

[0108] The present application can take the form of a computer program product including a program module accessible from a computer-usable or computer-readable medium storing program codes for use by or in connection with one or more computers, processors, or instruction execution systems. For the purposes of this description, a computer-usable or computer-readable medium is any apparatus that can contain, store, communicate, propagate, or transport the programs for use by or in connection with an instruction execution system, apparatus, or device. The medium itself can be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium because the signal carrier is not included in the definition of physical computer-readable medium including a semiconductor or solid state memory, magnetic tape, removable computer diskette, random access memory (RAM), read-only memory (ROM), a hard magnetic disk, and optical disk such as a compact disk read-only memory (CD-ROM), compact disk read / write, and DVD. Both the processor and the program codes for implementing each aspect of the present technology can be centralized or distributed (or a combination thereof) as known to those skilled in the art.

[0109] While the present application has been illustrated and described in detail in the preferred embodiments, the present application is not limited to the disclosed examples. Other variations can be derived by those skilled in the art without departing from the scope of the claimed application.

[0110] Reference List

[0111] 100 Environment of an apparatus for verifying a software-driven system

[0112] 105 Apparatus for verifying a software-driven system

[0113] 110 User device

[0114] 115 Network

[0115] 120 Processing unit

[0116] 125 Memory

[0117] 130 Storage unit

[0118] 135 Communication unit

[0119] 140 Network interface

[0120] 145 Input unit

[0121] 150 Output unit

[0122] 155 Standard interface or bus

[0123] 160 Test module

[0124] 165 pre-processing module

[0125] 170 scenario recognition module

[0126] 175 test case generation module

[0127] 180 evaluation module

[0128] 185 verification module

[0129] 190 report generation module

[0130] 195 database

Claims

1. A computer-implemented method for validating a software-driven system based on real-world scenarios in a computer-aided engineering environment, the method comprising: obtaining, from a source, a plurality of test scenarios corresponding to tests of the software driven system, by the processing unit (120) in a form defined with variable parameters, wherein, the source is one of an input device, a user device, or a database, and wherein the variable parameters include at least one of attributes and process parameters associated with the software-driven system; defining generation of a design space comprising a plurality of test scenarios based on the variable parameters, wherein the design space refers to multi-dimensional combinations and interactions of the variable parameters, wherein the design space contains possible combinations of variable parameters and each combination of the variable parameters corresponds to a test scenario or a design point, wherein the design space includes both feasible test scenarios suitable for testing the software-driven system and infeasible test scenarios unsuitable for testing the software-driven system; generating at least one real-world scenario associated with the software-driven system based on a first set of variable parameters, this step includes determining one or more constraints associated with the variable parameters and applying the determined one or more constraints to the design space to generate a pruned design space, the pruned design space corresponds to the at least one real-world scenario; identifying one or more feasible test scenarios suitable for testing the software-driven system based on the at least one real-world scenario from the plurality of test scenarios by sampling the pruned design space using a trained machine learning model, wherein the pruned design space is sampled using a machine learning algorithm to generate a number N of required samples, and wherein the identifying step further comprises the steps of: applying k-means clustering to the design space to generate k clusters and generating kxN random samples from the k clusters; applying constraints to the design space to determine an infeasible design space, wherein the infeasible design space corresponds to design points or test scenarios unsuitable for testing the software-driven system, wherein the remaining design points in the design space correspond to test scenarios suitable for testing the software-driven system; clustering the kxN random samples within the feasible design space using k-means clustering to form N clusters and computing centroids of the N clusters; checking each centroid of the N clusters to determine whether any centroid falls in the infeasible design space; if no centroid falls in the infeasible design space, generating the centroids of the N clusters as samples of feasible test scenarios for testing the software-driven system, otherwise repeating the steps of applying k-means clustering to the design space to generate k clusters and generating kxN random samples from the k clusters until a predefined termination criterion is met; determining whether the first set of test scenarios is suitable for testing the software-driven system based on a number of feasible test scenarios present within the first set of test scenarios; generating a simulation environment using a simulation tool in the computer-aided engineering environment, the simulation environment representing the real-world scenario in which the software-driven system is to be tested, wherein simulation agents in the simulation environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate; evaluating behavior of the software-driven system by applying the identified test scenarios to a model of the software-driven system in the simulated environment; and verifying behavior of the software-driven system in the real-world scenario based on results of the evaluation.

2. The method of claim 1, further comprising: generating a model of the software-driven system in the computer-aided engineering environment.

3. The method of claim 1, wherein, identifying one or more feasible test scenarios suitable for testing the software-driven system comprises: generating samples from the pruned design space by feeding inputs corresponding to the pruned design space to the trained machine learning model; and determining the one or more feasible test scenarios based on the generated samples, wherein each of the samples comprises values of the variable parameters to be used in a particular test scenario.

4. The method of claim 3, further comprising: generating an optimal sample from the pruned design space based on at least one optimization criterion; and determining the one or more feasible test scenarios based on the generated optimal sample.

5. The method according to any one of the preceding claims, wherein, evaluating behavior of the software-driven system by applying the identified test scenarios to a model of the software-driven system in the simulated environment comprises: generating simulation instances for testing the software-driven system based on the identified one or more test scenarios; executing the simulation instances based on the model of the software-driven system to generate simulation results; and analyzing the simulation results to determine behavior of the software-driven system in the real-world scenario.

6. The method of any one of claims 1-4, wherein, verifying behavior of the software-driven system in the real-world scenario based on results of the evaluation comprises: determining whether the behavior of the software-driven system in the real-world scenario meets an expected standard.

7. The method of any one of claims 1-4, further comprising: generating a notification on a graphical user interface indicating results of the verification.

8. An apparatus (105) for testing a software-driven system, the apparatus (105) comprising: one or more processing units (120); and a memory (125) communicatively coupled to the one or more processing units (120), wherein, the memory (125) comprises a test module (160) stored in the form of machine-readable instructions executable by the one or more processing units (120), wherein the test module (160) is configured to perform the method steps of any one of claims 1 to 7.

9. A cloud computing system comprising: one or more user devices (110); and the apparatus (105) of claim 8, the apparatus (105) being communicatively coupled to the one or more user devices (110), the user devices (110) being configured to verify a software-driven system according to any one of method claims 1 to 7.

10. A computer program product having machine-readable instructions stored therein, which, when executed by one or more processing units (120), cause the processing units (120) to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium comprising instructions which, when executed by one or more processing units (120), cause the one or more processing units (120) to perform the method according to any one of claims 1 to 7.

Images