Parallelized and asynchronous evaluation for optimization for maximum efficiency of information gain

Abstract

The present disclosure relates to a device for vehicle testing configured to perform operations including testing a first test sample batch including a plurality of first test samples to output a plurality of first test sample results, identifying one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed, calculating a parametric metric based on the one or more asynchronous test sample results, and adjusting a second test sample batch based on the parametric metric.

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of U.S. Provisional Application No. 63 / 366,770, filed on June 21, 2022, and U.S. Provisional Application No. 63 / 366,830, filed on June 22, 2022, the disclosures of which are hereby incorporated by reference in their entireties.

[0002] [Technical Field] The present disclosure relates to vehicle simulation, test, and verification systems and methods.

Background Art

[0003] Unless otherwise indicated herein, the subject matter described herein is not prior art to the claims of this application and is not to be considered prior art by virtue of being included in this section.

[0004] It is possible to test software using a simulated environment. For example, it is possible to test the software of an autonomous vehicle using a simulated driving environment. An autonomous vehicle can use sensors to perceive the environment of the autonomous vehicle. A control system can model the perception input from the sensors to determine a navigation route and make decisions in response to traffic controls such as stop signals, roundabouts, stop signs, speed limit changes, and other vehicles.

[0005] Vehicles may have various levels of automation. For some vehicles, the automation system may be able to provide warnings but may not otherwise control the vehicle. For other vehicles, the vehicle can operate on many different surfaces, seasons, and weather conditions without human intervention. As a result, devices, systems, and methods for testing, simulating, and verifying vehicles with a wide range of automation levels may be useful.

[0006] The subject matter claimed in this disclosure is not limited to examples that solve any disadvantages or that function only in an environment as described above. Rather, this background is provided only to illustrate one exemplary technical field in which some examples described in this disclosure may be practiced. SUMMARY OF THE INVENTION

[0007] A device for vehicle simulation, testing, and validation may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints defined using a first scenario format. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object configured to link to one or more objects using the internal parameter set. The processor may be configured to transmit the test configuration object to a test modality for testing.

[0008] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints defined using a first scenario format. The processor may be configured to map the first parameter set to an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0009] A non-transitory computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, enable a vehicle simulator to receive scenario format data, convert the scenario format data into data in an internal parameter format (internal parameter format data), generate a test configuration object using the data in the internal parameter format, and test the test configuration object using a test modality.

[0010] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first test sample batch that includes one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of one or more of the plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a parametric metric that includes one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second test sample batch based on the parametric metric.

[0011] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify one or more asynchronous test sample results of one or more of the plurality of first test samples before the first test sample batch completes testing. The processor may be configured to calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric. The processor may be configured to select a set of algorithms based on the parametric metric. The processor may be configured to apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch completes testing.

[0012] A computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, cause a vehicle tester to test a first test sample batch in a test modality format including one or more first test samples for outputting one or more first test sample results, identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed, convert the one or more asynchronous test sample results from the test modality format to an internal parameter format, and be capable of adjusting a second test sample batch based on the one or more asynchronous test sample results.

[0013] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric using the internal parameter set.

[0014] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints defined using a first scenario format. The processor may be configured to map the first parameter set to an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric based on one or more of a performance metric or a coverage metric.

[0015] Devices for vehicle simulation, testing, and validation may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first batch of test samples including one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of a plurality of the first test samples before the first batch of test samples is completed. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second batch of test samples based on the metric.

[0016] Devices for vehicle simulation, testing, and validation may include a memory and a processor operably coupled to the memory. The processor may be configured to identify first test data including one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data based on a first test modality. The processor may be configured to calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality. The processor may be configured to calculate second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data using the conversion function.

[0017] At least the elements, features, and combinations particularly noted in the claims achieve and realize the objectives and advantages of the examples.

[0018] Both the above overview and the following modes for carrying out the invention are given as examples and are illustrative, and do not limit the claimed invention.

Brief Description of the Drawings

[0019] Examples are described and explained with additional particularity and detail through the use of the accompanying drawings.

[0020]

Figure 1A

Figure 1B

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

DETAILED DESCRIPTION OF THE INVENTION

[0021] It is possible to test vehicle software (or autonomous vehicle software) using a vehicle software test bed, which may be an autonomous vehicle software test bed. The vehicle software test bed can simulate an operating environment for testing vehicle software (e.g., for determining how vehicle software performs in various driving situations). For example, the vehicle software test bed can simulate a three-dimensional (3D) geographical environment and provide inputs to the vehicle software that are similar to the inputs that the vehicle software could receive under real-world driving conditions, assuming the vehicle software is operating on a vehicle while navigating in an actual geographical environment.

[0022] Determining a set of scenarios to generate and execute in a simulation is useful for efficiently testing and validating a vehicle system, and more specifically, an autonomous vehicle (AV) system. Identifying a set of scenarios that sufficiently cover the test space to a statistical confidence level helps demonstrate the accuracy and safety of the underlying system under test.

[0023] Previous approaches to scenario generation and coverage calculation are based on three underlying assumptions: (1) a scenario definition language optimized for easy (but not necessarily comprehensive) parameterization, (2) contextualization and introspection into the simulation environment and stack, and (3) access to a closed-loop test environment. From these three assumptions, a verification engineer can define the parameters that make up the verification engineer's tests, the bad conditions for the tests, and the range of possible values and step sizes for each parameter. Combinations of parameters are then repeatedly tested, and new combinations are selected to gather additional information about the stack. Next, coverage calculation is based on which parameter values were tested across the entire set of tests. Coverage is calculated naively by binning the parameters into multidimensional histograms and clusters.

[0024] The limitations of these conventional approaches are that these approaches rely on the details of each of the above three assumptions. In particular, much previous work has relied on a dedicated scenario definition format, which may limit the expressiveness and extensibility of many aspects of the scenario compared to using an industry-standard scenario language. In addition, these scenario languages are often closely tied to the ability to introspect and modify these scenarios. Also, their closed-loop simulation environments for execution may not be realistic enough to provide accurate and realistic previous information for adaptively sampling. In addition to this, many of these algorithms rely entirely on previous samples collected before selecting the next sample to execute. Therefore, the ability of previous approaches for scaling is limited, and these approaches are mainly limited to the research context. Finally, naive approaches to computing coverage result in limitations in the ability to evaluate stack performance and test coverage in relation to how important different scenarios are.

[0025] Aspects of the present disclosure provide, among other things, systems and methods for enhanced simulation, scenario selection, and environment generation. For example, the systems and methods provide enhanced efficiency, such as through information sharing between different test modalities. The improved information sharing enables sampling to be performed more logically and efficiently.

[0026] Additional efficiency gains can result from the evaluation of the stack (e.g., performance and coverage), which may otherwise be a manual process, while the statistical interface can speed up the evaluation. Performance improvements resulting from parallelizable simulations and different test modalities can enable the generation of sampling algorithms that can converge faster. Also, using the disclosed techniques, AV tests can identify bad modes faster and statistically verify the autonomous stack.

[0027] Other aspects can provide a set of meta-statistics that quantify the amount of information loss that occurs based on this transformation when converting between different test modalities and risky / uncertain regions. The set of meta-statistics can quantify the amount of tests saved in each modality based on this correlation.

[0028] In yet another aspect, the disclosure includes methods for correlating information from software-in-loop (SIL), hardware-in-loop (HIL), and track tests to make predictions regarding performance in different situations. The aspect can provide the definition of a conversion function between the performance, coverage, or metric distribution of SIL, HIL, and track tests and the corresponding uncertainty.

[0029] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints defined using a first scenario format. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object configured to link to one or more objects using the internal parameter set. The processor may be configured to transmit the test configuration object to a test modality for testing.

[0030] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of a plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a parametric metric including one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second test sample batch based on the parametric metric.

[0031] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric using the internal parameter set.

[0032] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify first test data including one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data based on a first test modality. The processor may be configured to calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality. The processor may be configured to calculate second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data using the conversion function.

[0033] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0034] It is possible to map between open and / or custom representations and internal representations using a scenario-independent internal representation of the parameter space. It is possible to manipulate the parameter space itself to reduce the complexity and difficulty of the problems of search / optimization algorithms. Some exemplary data representations may include a scenario definition language (Automotive Open System Architecture (ASAM) Open Scenario (OSC) 1.0, OSC 2.0, or any other scenario language by a standardization body for automated and measurement systems), axes (variables and possible values of the variables, as well as output test metrics), relationships (defining how variables are related to each other), constraints (defining the boundaries of the axes and the limitations in the relationships), and the like.

[0035] As shown in FIG. 1A, a block diagram 100a for vehicle testing, simulation, and verification is provided. A set of parameters may be defined that includes mapping from any scenario language (e.g., an open scenario format such as OSC) or a proprietary scenario format such as a test specification for a HIL rig to an internal parameter format. That is, various scenario definition languages including SIL / HIL provided in OSC 2.0 as shown in block 105a, SIL / HIL as shown in block 105b, OSC 1.x, or a track (test case) as shown in block 105c may be mapped to a parameter space representation including one or more constraints, one or more values, or one or more relationships as shown in block 125.

[0036] Parallelized evaluation and / or asynchronous evaluation can provide maximum efficiency of information gain. Parallelized simulation can enable large-scale collection of information and generation of subsequent samples. Asynchronous evaluation can enable collection from hardware-in-the-loop (HIL) / on-vehicle test results to continue to notify the test. In some aspects, this technique may include a method for making decisions regarding further optimization using asynchronous information.

[0037] Thus, the simulation may be executed in a parallelized manner. The system can utilize the parallelized assumption to enable asynchronous updates of the sampling algorithm and previous ones for the parameter space. Specifically, the system may have the ability to enqueue new scenarios, build previous ones updated based on all the data collected so far, and use that previous one to enqueue the next scenario. Additionally, the integrated parameter space representation also enables the system to asynchronously update this parameter space performance using real-world data.

[0038] Using a parameter space representation as shown in block 125, it is possible to generate a specific scenario as shown in block 135. The specific scenario may be input into a parallelized test operation that includes various test modalities (such as SIL, HIL, track, etc.) as shown in block 145a. Results from the parallelized test (such as SIL, HIL, track, etc.) may be collected for different test modalities as shown in block 145b. It is possible to continue processes such as generating further specific scenarios and further parallelized tests by feeding back the results shown in block 145b to the parameter space representation as shown in block 125.

[0039] Various metrics may be used to evaluate coverage and performance across a scenario space. These scenarios may also be capable of enabling safety queries and audits in a quantifiable manner. And these aspects may be capable of enabling sampling algorithms to optimize parallelization in a way that maximizes statistical information metrics. Further, it is possible to describe performance and coverage using a single high-level statistically meaningful metric. The metric may include (a) an estimate of performance using confidence levels and confidence intervals that may normalize the integrated parameter space, and (b) an estimate of coverage based on sample density that may normalize the integrated parameter space, and (c) an integrated metric regarding the integrated parameter space, and a set of objective functions (e.g., boolean or continuous) and real-world observations.

[0040] Accordingly, the results shown in block 145b may be input into an evaluation metric operation, as shown in block 155. The evaluation metric operation may be configured to transform test results using a parameter space representation in order to generate a performance / coverage metric as described.

[0041] Information from SIL, HIL, and track tests can be correlated to make inferences about performance in different situations. A conversion function between the performance, coverage, or metric distributions of SIL, HIL, and track tests and the corresponding uncertainties may be determined. A set of meta-statistics may be capable of quantifying the amount of information loss during conversion between different test modalities and risky / uncertain regions based on this conversion. The set of meta-statistics may be capable of quantifying the amount of tests preserved in each modality based on this correlation.

[0042] An exemplary process flow 100b for vehicle simulation, testing, and verification is provided in FIG. 1B. The scenario cross-compiler 110 may be configured to map from a scenario language (e.g., an open scenario format such as OSC, or a proprietary scenario format such as a test specification for a HIL rig) to an internal parameter format (e.g., an intermediate YAML format). The internal parameter format may be provided to a sampling service 120 configured to store and / or process one or more axes, one or more relationships, or one or more constraints calculated using the internal parameter format. The linker service 130 may be capable of creating a test configuration object that links between different objects, test results, etc. using the internal parameter format.

[0043] The linker service 130 may be configured to provide the test configuration object to a block 140 that includes one or more of a queue service, a parallel job scheduler, or an intelligent job manager. The queue service may be capable of submitting the test configuration object to a plugin or service based on the satisfaction of a specific constraint. The parallel job scheduler may be capable of scheduling jobs for different test modalities, simulators, etc. The intelligent job manager may be capable of determining different jobs to test based on sampling of a probability distribution.

[0044] Test results resulting from the combined operation of the queue service, parallel job scheduling, and intelligent job manager may be provided to block 150, which includes post-processing services, sampling services, data indexing, and feature quantification. The post-processing service may spin down software. The sampling service, data indexing, and feature quantification can group primary indexes with filtered values based on feature vectors. Data from block 150 may be provided to block 160, which includes inference services and distribution post-processing. The inference service can filter related test-independent data into a multi-dimensional distribution. The distribution post-processing can generate an annotated data point probability distribution using the data point probability distribution.

[0045] The annotated data point probability distribution may be provided to block 170, which may include various other probability distributions (data point probability distributions specific to extended test modalities, annotated data probability distributions specific to test modalities). The probability distribution from block 170 may be provided to a learning correlation block 180 configured to generate a conversion function 190. The conversion function 190 may be configured to convert between different test modalities or other distributions. The data point probability distribution specific to the extended test modality may be provided to block 140 to continue the test process.

[0046] Internal representation A variety of industry standard scenario formats or custom formats may be used for the simulation, testing, and verification of vehicles such as autonomous vehicles. A scenario-independent internal representation of the parameter space (e.g., an internal parameter format) and a mapping between this internal representation (e.g., an internal parameter format) and an open standard enable a mapping between a variety of industry standard scenario formats or custom formats and this internal representation (e.g., an internal parameter format). The internal representation of the parameter space (e.g., an internal parameter format) can enable the inclusion of information across one or more test modalities to facilitate additional information regarding the coverage and performance of the parameter space.

[0047] The internal representation of the parameter space (e.g., an internal parameter format, etc.) may include one or more components including (i) one or more axes that may include one or more variables and possible values for the one or more variables, (ii) one or more relationships that may include one or more relationships between the one or more variables for the one or more axes, and combinations between the one or more variables for the one or more axes, and (iii) one or more constraints that can define one or more boundaries of the variables among the one or more variables and define other regions within the scenario space that may be invalid.

[0048] Using this internal representation of the parameter space (e.g., internal parameter format, etc.), a variety of statistical operations may be performed in a scenario - and test - modality - independent manner. In particular, various statistical operations such as multidimensional clustering, nearest - neighbor analysis, parametric (or non - parametric) and / or linear (or non - linear) regression are performed to inform the sampling algorithm more effectively compared to baseline sampling algorithms that do not use these various statistical operations. In particular, using various statistical operations, optimization algorithms and search algorithms can mutate the abstracted parameter space in a reversible manner so that they can operate on the search space more efficiently (e.g., with respect to computational complexity) compared to the parameter space where these various statistical operations are not performed. In particular, the application of these statistical operations can facilitate one or more of (a) dimensionality reduction, (b) conversion from a non - convex search space to a convex search space, or (c) increasing the connectivity and / or continuity of the scenario space when compared to the parameter space where these various statistical operations are not performed.

[0049] An internal representation of a parameter space (e.g., an internal parameter format) may be used by a device operable for one or more of vehicle simulation, vehicle testing, or vehicle verification. The device may comprise a memory and a processor operably coupled to the memory. The processor may be configured to execute instructions for causing the device to identify a first parameter set defined using a first scenario format. The first parameter set may include one or more of (i) one or more first scenario format axes, (ii) one or more first scenario format relationships, or (iii) one or more first scenario format constraints. The processor may be configured to execute instructions for causing the device to map the first parameter set to an internal parameter set using an internal parameter format, the internal parameter format including one or more of (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints. The processor may be configured to execute instructions for causing the device to generate a test configuration object using the internal parameter set. The test configuration object may be configured to link to one or more objects. The processor may be configured to execute instructions for causing the device to send the test configuration object to a test modality for testing.

[0050] A process flow 200 for vehicle simulation, testing, or verification may be provided as shown in FIG. 2. Inputs (such as test specification 210) related to a scenario in a first scenario format (such as a scenario definition language) may be identified. The test specification 210 may include one or more of a hardware-in-the-loop (HIL) test specification, a software-in-the-loop (SIL) test specification, or a field test specification. The HIL test specification may include, for example, a dSpace® HIL specification. The SIL test specification may be implemented using a first scenario format (such as a scenario definition language) such as OSC1.0, OSC2.0, different scenario languages, etc.

[0051] In one example, if the scenario is outside a particular scenario language, the scenario may be imported or converted into the particular scenario language. If it is in the particular scenario language, the test specification 210 may be cross-compiled into an internal parameter format that may be in an intermediate yet another markup language (YAML) format as shown in operation 220 that can be performed in a one-to-one operation. The internal parameter format may include one or more of (i) one or more internal axes, (ii) one or more internal relationships, (iii) one or more internal constraints, (iv) one or more internal metrics, or (v) one or more parse operation information that facilitates inverse conversion to a first scenario format (such as a particular scenario language).

[0052] In another example, if the scenario is in a particular scenario language, the internal parameter format may be configured to convert from the particular scenario language to the internal parameter format without importing or converting the scenario into the particular scenario language. The internal parameter format may include additional conversion data that may be used to facilitate the conversion from the particular scenario language to the internal parameter format.

[0053] As a result, cross-compiling the test specification 210 into the first parameter set of the internal parameter format may cause one or more internal axes, one or more internal relationships, or one or more internal constraints to be parsed into the internal parameter format, as shown in operation 230 which can be performed in a one-to-one operation. The internal parameter format (e.g., YAML format test specification, etc.) may be exchangeable between various test modalities (e.g., HIL, SIL, field test, etc.) and simulators (e.g., HIL simulator, SIL simulator, field test simulator, etc.). This internal parameter format may include one or more parsing operation information (e.g., related links and context information, etc.) used to identify the compilation source of the internal parameter format in order to facilitate reverse tracing of the internal parameter set of the internal parameter format to the first parameter set of the first parameter format (e.g., a specific scenario language, etc. that may be specific to a specific scenario language). The internal parameter format may also include one or more internal metrics for generating metrics using axes when the results are available after the test.

[0054] It is possible to generate one or more of a test configuration object, a test run specification, or a test run result using the first parameter set of the internal parameter format. It is possible to generate a test configuration object (i.e., the test configuration object), as shown in operation 240 which may be a one-to-many operation, using the first parameter set of the internal parameter format, for example, an intermediate YAML format. The test configuration object may include references to one or more of a test specification, a parameter set of the internal parameter format, a test result, etc. to facilitate indexing of data related to a specific test.

[0055] To start a test, a request (and related links) containing the first parameter in the internal parameter format is sent to the microservice as shown in operation 250, which may be a one-to-many operation, to enqueue a test job and generate a test run specification. The test run specification may be a data package containing information used to execute the test in a test environment. This information that may be used to execute the test in a test environment may include (i) a test environment specification, (ii) references to one or more input files such as a test specification file, and (iii) one or more of runtime parameters, etc. The test configuration object may be formatted in a test modality format (such as a format specific to SIL, HIL, track, etc.) for a test modality (such as SIL, HIL, track, etc.). The test configuration object to be formatted may be sent to a test modality (such as a SIL test, HIL test, track test, etc.) associated with the test modality (such as SIL, HIL, track, etc.).

[0056] Requirements for generating test run specifications based on the required test modalities may be transferred to one or more interfaces for one or more test modalities, including (i) an interface to a specific test scenario simulator, (ii) a plugin interface to one or more of an external SIL simulator or an external HIL simulator, or (iii) a plugin interface to an automated track test job queue. These one or more interfaces may perform further formatting from an internal parameter format (such as an intermediate scenario language) to the test specification. The further formatting may include one or more of a HIL rig specification, a track test specification, a SIL specification, etc. These interfaces (such as a plugin interface) may submit the requirements to the corresponding test modality (such as a SIL test modality for a SIL test specification, a HIL test modality for a HIL test specification, a track test modality for a track test specification, etc.) via an available web application protocol interface (API) (such as Representational State Transfer (REST), Hypertext Transfer Protocol (HTTP), etc.).

[0057] As shown in operation 260, the test run specification operation 250 may be capable of generating a test run result, which may be a one-to-one operation. The test run result may include one or more of drive data, simulation data, or additional data that may be parsed based on an axis type.

[0058] Test modality format data may be converted into data in the internal parameter format in an internal parameter format converter plugin. In one example, external test run results (such as test run results for a particular scenario language) may be received using a converter plugin. The external test run results may include various test modalities including one or more of SIL, HIL, drive, etc. The converter plugin can receive various inputs including (i) the internal parameter format, (ii) drive logs in an arbitrary format, etc. The converter plugin can define a conversion from the drive log format to the internal parameter format. Metrics in the internal parameter format may be identified after testing.

[0059] Using the test run results, it is possible to generate test-independent data points using a one-to-many operation as shown in operation 270. The test run results, which may include drive data, simulation data, etc., may exist in a time-based metric format. The time-based metric format may be collected from various test modalities such as SIL, HIL, drive, track, etc. The time-based metric format may be converted into an internal parameter format (such as an intermediate YAML format, for example). The time-based metric format may be converted into an internal parameter format based on one or more of (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints.

[0060] The internal parameter format converter plugin may be configured to convert axis data in a time-based metric signal format into data in an internal parameter format and write the axis data to a database. In one example, it is possible to use a YAML template for an axis to convert the time-based metric format into an intermediate YAML format and generate an intermediate YAML format metric. The post-processing microservice can calculate the relevant axis values from these intermediate YAML format metrics that can be written to the database. Any axis that violates the constraints or relationships specified in the intermediate YAML format may be updated in the intermediate YAML format to include input axis values that may be identified as untested.

[0061] It is possible to generate a data point probability distribution as shown in operation 280, which may be a many-to-many operation, using test-independent data points that may be generated as shown in operation 270. The data point probability distribution may be generated using one or more of (i) the axes referenced in a particular data point probability distribution, (ii) the values referenced in a particular data point probability distribution, or (iii) the parameters referenced in a particular data point probability distribution.

[0062] The scenario definition language may include various scenario languages (such as open scenario formats like OSC, or proprietary scenario formats such as test specifications for HIL rigs). In one example, the scenario definition language may be an open scenario language. When the open scenario language is used, a simulator (such as an autonomous vehicle (AV) simulator) may be configured to receive scenario format data and convert the scenario format data into data in an internal parameter format (such as intermediate YAML format data as shown in operation 220). The vehicle simulator may be configured to generate test configuration objects using data in the internal parameter format (such as shown in operation 240) and test the test configuration objects using a test modality (such as shown in operation 250). The vehicle simulator may be configured as otherwise disclosed with respect to generating an internal representation of the parameter space provided herein.

[0063] The vehicle simulator may be configured to generate various metrics (such as shown in operation 260) for measuring one or more of the coverage or performance of the vehicle simulator. The vehicle simulator may be configured to calculate parametric metrics (such as metrics calculated in a parametric format). The parametric metric may be based on one or more of a parametric performance metric or a parametric coverage metric. The parametric metric may be selected based on a particular use case type. The vehicle simulator may be configured to extract one or more axis values for a test modality and calculate one or more axis-specific metrics based on the one or more axis values. The vehicle simulator may be configured to generate test configuration objects based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric.

[0064] Parallel Simulation Using Asynchronous Information The functionality for vehicle simulation, testing, or verification may be configured to generate parallel simulations using asynchronous information. Combining asynchronous information with parallelized simulations can increase the information gain over a specific period of time for a selected amount of computing resources. The parallelized simulations can facilitate the large-scale collection of information and the subsequent generation of samples. The update of asynchronous information can facilitate the collection from various test modalities (e.g., SIL, HIL, track, etc.) for determining subsequent tests. It is possible to update the currently running parallel tests using partial information without random scheduling of subsequent test jobs and without waiting for complete information before adjusting subsequent test jobs. Therefore, tests that combine parallel job scheduling with asynchronous information can increase the information gain compared to the baseline approach.

[0065] In addition, adding, adjusting, or terminating different test samples based on asynchronous information before the completion of tests for a specific test batch can reduce the computational complexity for achieving convergence for a specific test case compared to the baseline computational complexity where asynchronous information is not used, parallel tests are not used, or neither asynchronous information nor parallel tests are used. The reduction in computational complexity for achieving convergence for a specific test case may occur due to an increase in the number of tests for one or more of the high-sensitivity cases, undetermined cases, edge cases, or cases where defects are likely to occur, and a decrease in the tests for cases with lower information compared to the baseline test cases. For example, cases with lower information may include low-sensitivity cases, high-coverage cases, redundant test cases, cases with appropriate test coverage, low-defect cases, etc.

[0066] The sampling algorithm and operations on the parameter space may be updated asynchronously using parallelized assumptions. That is, when a scenario is queued, it is possible to generate an updated previous one based on the collected data and queue subsequent scenarios. A scenario may include data collected asynchronously using an internal parameter format. The data collected asynchronously may include one or more of simulated data or real-world data.

[0067] The vehicle simulator may be configured for one or more of simulation, testing, or verification as provided in process flow 300a in FIG. 3A. The vehicle simulator may be configured to test a first test sample batch in a test modality format. The first sample batch may include one or more first test samples for outputting one or more first test sample results. The vehicle simulator may be configured to identify one or more asynchronous test sample results of a plurality of first test samples before the first test sample batch is completed. The vehicle simulator may be configured to convert one or more asynchronous test sample results from a test modality format to an internal parameter format. The vehicle simulator may be configured to adjust a second test sample batch based on one or more asynchronous test sample results.

[0068] The vehicle simulator may be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (such as, for example, a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved. For example, after testing the first and second batches, the coverage metric may not have achieved the selected threshold. In this example, the vehicle simulator may be configured to adjust subsequent test sample batches (such as, for example, the third, the fourth, etc.) until the coverage metric achieves the selected threshold. Testing of subsequent test sample batches may be repeated until the selected coverage metric is achieved.

[0069] The vehicle simulator may be configured to process test specifications for transmission to the queue service 340a using one or more operations. The test specifications may include one or more of, as shown in block 302, HIL test specifications, OSC™ 1.0 or OSC™ 2.0 test specifications, field test specifications, etc. The test specifications may be provided to a scenario cross-compiler 310 that is capable of compiling the test specifications into an internal parameter format (such as, for example, an intermediate YAML representation, etc.), as shown in block 312.

[0070] The internal parameter format (e.g., intermediate YAML representation, etc.) may be provided to one or more of the sampling service 320 or the linker service 330. As shown in block 322, the sampling service 320 can generate one or more object representations of one or more axes (e.g., numerical axis or discrete axis, etc.), one or more relationships, or one or more constraints. The linker service 330 can receive the internal parameter format (e.g., YAML representation, etc.) as shown in block 332 and generate a test configuration object. The test configuration object as shown in block 332 may be provided to the queue service 340a.

[0071] As shown in block 332, the queue service 340a can receive a test configuration object and, as shown in block 338, can receive user input (e.g., user requirements, etc.). The queue service 340a can submit the test configuration object as shown in block 332 and the user input as shown in block 338 to one or more of the plugins or services for execution based on the satisfaction of one or more constraints. The test configuration object as shown in block 342 may be provided to one or more of the HIL plugin 344a, the SIL plugin 344b, the field test plugin 344c, or the test run specification 346.

[0072] The queue service 340a is capable of receiving a test trigger 348 from the intelligent job manager 340c. The intelligent job manager 340c may be configured to sample a data probability distribution (such as an extended data probability distribution) for one or more test cases for a particular test modality (such as SIL, HIL, track, etc.). The one or more test cases may be one or more of a sensitive case, an indeterminate case, an edge case, or a case where a defect is likely to occur. A sensitive case may be a test case based on a variable that can have a large impact on the result when compared to a baseline test case. The baseline test case may be a test case that can be based on one or more of an average test sample result, a median test sample result, etc. An indeterminate case may be a test case that does not contain sufficient test sample results to provide sufficient information regarding coverage, performance, or other suitable metrics. An edge case may be a test case that tests the boundary between one or more different test cases. A case where a defect is likely to occur may be a test case that is more likely to result in a defect when compared to a baseline test case.

[0073] The Intelligent Job Manager 340c may be configured to identify previous test sample results, currently executing test samples (e.g., test jobs, simulations, etc.), and the overall probability space of different test cases. The Intelligent Job Manager 340c may be capable of receiving test sample results that can change the data probability distribution. The test sample results may include asynchronous test sample results. The Intelligent Job Manager 340c may be configured to receive and / or identify asynchronous test sample results from a first batch of test samples in a specified test modality format before the first batch of test samples completes the test. The Intelligent Job Manager 340c may be configured to convert the asynchronous test sample results into an internal parameter format when the asynchronous test sample results are in a test modality format (e.g., SIL, HIL, track, etc.).

[0074] The Intelligent Job Manager 340c may be configured to adjust a second batch of test samples based on one or more asynchronous test sample results. For example, the Intelligent Job Manager 340c may cancel the currently executing test samples in the first batch of test samples and release computing resources for other tests when the unique test sample results associated with the currently executing test samples are not used to change the data probability distribution based on asynchronous test sample results that can change the data probability distribution. The cancellation request may be sent from the Intelligent Job Manager 340c to the parallel job scheduler 340b (as shown in block diagram 300b in FIG. 3B).

[0075] The intelligent job manager 340c may be configured to down-weight current test samples when sampling a data probability distribution in order to maximize information gain and minimize redundant tests. The current test samples may not have test sample results included in the data probability distribution, but the current test samples may have been previously enqueued. Thus, to avoid double counting current test samples, the data probability distribution may be adjusted based on the current test samples (e.g., such as being down-weighted).

[0076] The vehicle simulator may be configured to process test specifications for generating test run results 358 using one or more operations, as shown in FIG. 3B. Test run specifications 346 (such as provided by one or more of queue service 340a, HIL plugin 344a, SIL plugin 344b, or field test plugin 344c) may be provided to parallel job scheduler 340b.

[0077] The parallel job scheduler 340b may be configured to (i) test one or more test sample batches in various test modality formats and (ii) send the test sample batches to various test modalities. The test sample batches may include one or more test samples that may be calculated to output one or more test sample results.

[0078] The parallel job scheduler 340b may be configured to identify various parameters related to job scheduling, including one or more of randomness, decreasing learning rate, predicted sample size, test batch configuration parameters (such as hyperparameters related to past, current, and predicted future test executions). The parallel job scheduler may be implemented using various algorithms that may include hyperparameters based on the operating parameters of the algorithms.

[0079] The parallel job scheduler 340b is configured to select a test sample batch and provide the test sample batch in the test run specification 352 to one or more of one or more test modality plugins (such as the HIL plugin 354a, the SIL plugin 354b, the field test plugin 354c, etc.) for formatting into a test modality format (such as HIL, SIL, field test, etc.) that is input into a test modality format (such as the HIL simulation 356a, the SIL simulation 356b, the field test 356c, etc.), or (ii) as an input for a specific simulation 356d test. Results from one or more of the HIL simulation 356a, the SIL simulation 356b, the field test 356c, the applied simulation 356d test, etc. may be collected as test run results 358. These test run results may be provided to the job post-processing service 450a (as shown in FIG. 4A).

[0080] The parallel job scheduler 340b may be configured to schedule one or more of jobs, simulations, test batches, etc. to different test modalities, simulators, etc. by allocating software, network, or other computing infrastructure for operating jobs, simulations, test batches, etc. in parallel. The intelligent job manager 340c may be configured to send requests 359 to the parallel job scheduler 340b to intersect and cancel jobs to free up computing resources. The intelligent job manager 340c may be configured to generate a test specification (such as a newly generated test specification 311) that is processed as a test specification by being input into a scenario cross-compiler 310 (as shown in FIG. 3A).

[0081] The parallel job scheduler 340b may be configured to receive requests 359 to intersect and cancel jobs in order to free up computing resources. Requests 359 to intersect and cancel jobs in order to free up computing resources may be based on asynchronous test sample results. As a result, the parallel job scheduler 340b can adjust subsequent test sample batches based on the asynchronous test sample results. Alternatively, or in addition, the parallel job scheduler 340b may be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (such as a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0082] The test run results 358 may include various metrics, axis values, etc. The test run results 358 may include one or more of a stack log or an environment log. The test run results 358 may include various data files such as sensor and rendering data files. The test run results 358 may be provided to the post-processing service 350a. The post-processing service 350a may be configured to convert axis data in a time-based metric signal format into data in an internal parameter format in an internal parameter format converter plug-in. The axis data, which may be in a time-based metric signal format or an internal parameter format, may be written to a database. The database may be used when retrieving one or more axis values for a test modality (such as SIL, HIL, track, etc.) and when calculating one or more axis-specific metrics based on the one or more axis values. Storing data in an internal parameter format can facilitate efficient data retrieval when performing calculations. Alternatively, or in addition, calculations may be vectorized to improve computational efficiency.

[0083] It is possible to process the test run result 358 with various operations to generate a data probability distribution to be sent to the intelligent job manager 340c. The data probability distribution may be calculated using parametric metrics (such as metrics based on one or more parameters). The parametric metric may be calculated using one or more of a parametric performance metric or a parametric coverage metric. The parametric metric may be based on a specific use case type.

[0084] Parametric metrics (such as tunable survey metrics, etc.) may be used to survey the information space to be tested. In one example, the parametric metric may be tuned based on the difference between the random sample and the collected sample. For example, for a uniform discrete distribution from 1 to 10, a heavy weighting of samples near the upper end of the range may indicate that samples near the lower end of the range may also be surveyed. The tunable survey metric may be adjusted when samples near the lower end of the distribution are collected. That is, using the tunable survey metric, the degree of samples outside the distribution can be determined and may be adjusted when the distribution changes in response to sample collection.

[0085] Alternatively, or in addition, the parallel job scheduler 340b may be configured to generate subsequent test sample batches based on one or more axis-specific metrics. The one or more axis-specific metrics may be based on one or more axis values selected based on parametric metrics.

[0086] The parallel job scheduler 340b may be configured to schedule jobs using asynchronous information (such as, for example, asynchronous test results). The parallel job scheduler 340b may be configured to schedule one or more different test samples in parallel and to use asynchronous information (such as, for example, asynchronous test results) to end, adjust, or add different test samples. The parallel job scheduler 340b may be able to end, adjust, or add different test samples using asynchronous information (such as, for example, asynchronous test results) without having a completion time for a particular test sample. The parallel job scheduler 340b may be able to receive additional test samples for testing based on asynchronous information (such as, for example, asynchronous test results) before a previous test sample has completed the test.

[0087] A device for testing, simulating, or validating a vehicle may comprise a memory and a processor. The processor may be configured to execute instructions for causing the device to test a first test sample batch including one or more first test samples and output one or more first test sample results. The plurality of first test sample results may include completed test samples (which may be used as asynchronous information) and uncompleted samples (which may thus not be used as asynchronous information). The plurality of first test sample results may be generated using one or more different test modalities (such as, for example, SIL, HIL, track, etc.).

[0088] The processor may be configured to identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed. That is, the asynchronous test sample results may be either incomplete test sample results of the first test sample batch or results of completed test samples. The asynchronous test sample results may be based on real-world data. It is possible to calculate parametric metrics using one or more asynchronous test sample results.

[0089] The parametric metric may include one or more of a parametric performance metric or a parametric coverage metric. The first sample batch may be terminated prior to completion based on the parametric metric. It is possible to adjust a second test sample batch using the parametric metric. The second test sample batch may be adjusted based on test adjustment configuration parameters including one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, and the like.

[0090] The parallel job scheduler 340b may be configured to convert a plurality of first test samples from an internal parameter format to a test modality format. For example, the internal parameter format may be converted to a HIL format (e.g., using a HIL plugin), to a SIL format (e.g., using a SIL plugin), or to a field test format (e.g., using a field test plugin).

[0091] The plurality of first test samples may be in a test modality format when asynchronous information is identified. Asynchronous information (e.g., one or more asynchronous test sample results, etc.) may be converted from the test modality format to the internal parameter format (if present in the test modality format). Converting the asynchronous information to the internal parameter format makes it possible to minimize the computational complexity, computational time, and cost of adjusting the second test sample batch based on the asynchronous information.

[0092] Metric A device for vehicle simulation, vehicle testing, or vehicle verification may include a memory and a processor operably coupled to the memory. The processor may be configured to execute instructions to cause the device to identify a first set of parameters defined using a first scenario format. The first set of parameters may include one or more of one or more first scenario format axes, one or more first scenario format relationships, or one or more first scenario format constraints.

[0093] As shown in FIG. 4A, functionality 400a may be provided in which a post - processing service 450a is configured to identify a first set of parameters. The post - processing service 450a may be configured to map the first set of parameters to an internal set of parameters. The internal set of parameters may include one or more of one or more internal axes, one or more internal relationships, or one or more internal constraints. The post - processing service may be configured to transform the first set of parameters (e.g., test run results, etc.) to generate an internal set of parameters (e.g., test - independent data point 451, etc.). The internal set of parameters (e.g., test - independent data point 451, etc.) may be input to a sampling service 450b.

[0094] The sampling service 450b may be configured to calculate asynchronous information provided to an intelligent job manager 340c (such as that shown in FIG. 3A for example). The sampling service 450b may be configured to identify one or more asynchronous test sample results of one or more first test samples before a first test sample batch finishes testing. The sampling service 450b may be configured to calculate parametric metrics. The parametric metrics may be based on one or more of parametric performance metrics or parametric coverage metrics. The sampling service 450b may be configured to select a set of algorithms based on the parametric metrics. The sampling service 450b may be configured to apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch finishes testing.

[0095] The sampling service 450b may be configured to generate and adjust an information space that may include an internal parameter set by adding a data index based on one or more of (i) one or more internal axis values, (ii) one or more internal relationship values, or (iii) one or more internal constraint values.

[0096] The sampling service 450b may be configured to select one or more of a set of axes based on parametric metrics or a subset of a set of axes. Results based on one or more of a set of axes based on parametric metrics or a subset of a set of axes may be displayed on a display device. One or more axis values for the set of axes or the subset of the set of axes may be retrieved for a particular test modality (such as HIL, SIL, track, etc.), and one or more axis-specific metrics may be calculated based on the set of axes or the subset of the set of axes.

[0097] The sampling service 450b may be configured to calculate parametric metrics. The parametric metrics may be based on one or more of parametric performance metrics or parametric coverage metrics. The parametric metrics may be based on a specific use case type. The sampling service 450b may be configured to generate test configuration objects based on one or more axis-specific metrics. The one or more axis-specific metrics may be based on one or more axis values selected based on the parametric metrics. Results based on the parametric metrics may be displayed on a display device.

[0098] As shown in FIG. 4B, various operations 400b indicate that it is possible to send test-independent data points to a data indexing block 460a that may be configured to process the test-independent data points 461 by having the sampling service 450b construct a primary index (as shown in block 462) and construct a secondary index (i.e., view) (as shown in block 464). The primary index may include one or more of (a) date, (b) axis, (c) test environment, (d) stack version, (e) other metadata, etc. (as shown in block 463).

[0099] The secondary index of block 464 that constructs the secondary index may be activated based on user input or based on the availability of computing resources. Block 465 that constructs the secondary index may be capable of generating a group of primary indexes that include filtered values configured to select specific values, ranges, etc. of the test-independent data points.

[0100] The test-independent data point 461 may be sent to the feature quantification block 460b that can process the test-independent data point 461 in a feedback loop to generate results transmitted to one or more of (i) the test-independent data point 461, block 463, or block 465. As shown in FIG. 4C, the further operation 400c may include sending the test-independent data point 461 to the feature quantification block 460b that may input it to a block 466 that applies a kernel function. It is possible to apply the kernel function to the test-independent data point to generate a feature vector 468. Various techniques can be used to generate the feature vector 468 including one or more of (i) linear, (ii) non-linear, (iii) deep learning-based functions, (iv) frequency space transformation, (v) other transformations, etc., as shown in block 467. The output of the feature quantification block 460b (such as the feature vector 468, etc.) may be sent to the data indexing block 460a (such as the test-independent data point 461, block 463, or block 465, etc.).

[0101] As shown in FIG. 4A, the sampling service 450b can use data indexing in block 460a and feature quantification 460b to generate test-independent data points and index 453. The test-independent data points and index 453 may be sent to the inference service 450c. The inference service 450c may be configured to filter the test-independent data points into a multi-dimensional distribution. The test-independent data points and index 453 may be filtered into the multi-dimensional distribution based on one or more of relevance or on various data point spaces that can generate a data point probability distribution 480 using one or more primary indexes, secondary indexes, feature amounts, etc. The data point probability distribution 480 may be sent for distribution post-processing 460c.

[0102] A user interface may be provided to enable a user to view at a glance one or more of vehicle stack performance, test coverage, or uncertainty / test gap via a web-based viewer. Through the user interface, the user can select any axis or combination of axes (such as performance metrics like pass rate, collision margin time, variables like weather, driving speed, road type, etc.). A multi-dimensional chart of test results related to the set of axes may be displayed via the user interface (such as a scatter plot matrix, cluster scatter plot, surface / contour plot, etc.) based on the selection on the user interface. Through the user interface, filters and switch axes can be applied to these charts to iteratively filter the information space (such as stack performance, test coverage, uncertainty / test gap, etc.).

[0103] In some examples, the user may click on recommendations for coverage gaps, highly correlated axes, relationships, clusters, and high uncertainty and low metric values. The user can select a combination of one or more of axes for viewing, grouping, clustering, smoothing, interpolation, or statistical metrics by sending a request to the sampling service 450b. The sampling service 450b can attempt to retrieve axis values for a particular test modality (such as SIL, HIL, track, etc.). When there are no axes for a particular test modality, the axis values can be calculated as requested, and the sampling service 450b can provide one or more of vehicle stack performance, test coverage, test gap, etc. to the user interface. Alternatively, or in addition, the sampling service 450b can provide one or more of vehicle stack performance, test coverage, test gap, etc. to the user interface when there are axes for a particular test modality.

[0104] When axis values are calculated, the distribution post - processing 460c operation may be configured to facilitate various operations for sending data to the probability distribution 470 operation and receiving the data point probability distribution 480, as shown in operation 400d illustrated in FIG. 4D. As shown in block 469a, it may be possible to expand the data point probability distribution 480 with a kernel function, or as shown in block 469b, it may be possible to calculate a derived distribution using the data point probability distribution 480.

[0105] When the data point probability distribution 480 is expanded with a kernel function, a data point probability distribution 469c may be calculated. When a derived distribution is calculated using the data point probability distribution 480, one or more of the gradient 469d or the density 469e may be calculated. As shown in block 469f, it is possible to perform a statistical analysis using one or more of the data point probability distribution 469c, the gradient 469d, or the density 469e.

[0106] The statistical analysis performed may include any suitable technique for generating a probability distribution including one or more of (i) clustering, (ii) regression, (iii) significance (e.g., based on a combination of coverage and / or performance), (iv) sensitivity analysis, or (v) entropy, as shown in block 469g. The distribution post - processing 460c operation may be capable of providing an input to the probability distribution 470 operation for further processing to the intelligent job manager 440c.

[0107] The statistical analysis performed may include any suitable analysis technique for analyzing the probability distribution of data points. The statistical analysis performed may include grouping combinations of axis values for further calculations based on one or more of metadata tags, scenarios, drive tags, locations, etc. Alternatively, or in addition, derived metrics (such as reliability, confidence interval, mean, minimum, maximum, count, standard deviation, statistical entropy, etc.) may be calculated. Alternatively, or in addition, clustering and correlation analysis may be performed to determine patterns. Exemplary patterns may include linear kernel separators, kernel-based correlation analysis, K-means, DB-SCAN clustering, etc. Alternatively, or in addition, outliers, clusters, and associated metrics may be linked to these results. Alternatively, or in addition, smoothing and interpolation may be performed to include one or more of Gaussian smoothing, linear interpolation, nearest neighbor interpolation, cubic interpolation, quintic interpolation, etc.

[0108] The user interface is capable of providing page filtering for specific scenarios, simulations, drives, etc. Inputs to the user interface may be submitted to a scenario cross-compiler 310, as shown in FIG. 3A, and compiled into scenarios and test cases of a specific test modality (such as HIL, SIL, track, etc.). A plug-in specific to the test modality may be configured to compile an internal parameter set (in an internal parameter format) into a test modality format that may include a basic external scenario.

[0109] As shown in block diagram 500 in FIG. 5, the output from the distributed post - processing block 560c may be sent to a probability distribution block 570 (such as, for example, the annotated data probability distribution block 571). The annotated data probability distribution block 571 may be a data probability distribution that may be used, for example, to calculate a metric using an internal parameter set. This metric may include one or more of a performance metric or a coverage metric. This metric may be based on the use - case type. This metric may be calculated using one or more of a confidence level, a confidence interval, a normalized format, or a sample density.

[0110] The annotated data probability distribution block 571 may be configured to identify a particular driving situation based on one or more of (i) an importance (such as, for example, sensitivity), (ii) a density with respect to the importance (such as, for example, being able to use further data collection), (iii) a correlation variable (such as, for example, regression), (iv) a clustering (such as, for example, a discrete pattern). It is possible to select an axis for additional testing using these metrics (such as, for example, importance, density with respect to importance, correlation variable, clustering).

[0111] A metric may be an integrated metric that is calculated based on performance metrics and coverage metrics. The integrated metric may be based on one or more of an objective function (e.g., boolean or continuous) or real-world data. The integrated metric may be a single metric that facilitates determination of performance and coverage for an integrated parameter space. That is, the integrated metric may be used for one or more validations or audits of one or more of performance or coverage for the integrated parameter space. In one example, a query may include a query such as, in response to a query of "Does my stack prevent collisions in hard braking events with a 99.99% success rate and a 0.01% error margin?", a set of statistical metrics (e.g., metric, integrated metric, etc.) may be returned that accept or reject the query.

[0112] Vehicle safety characteristics may be queried and / or audited using quantification. Using the annotated data probability distribution block 571, it is possible to query vehicle safety characteristics and calculate a safety metric based on a metric in response to a vehicle safety characteristics query.

[0113] It is possible to calculate a metric using parallelization to maximize one or more of an operational design domain (ODD) coverage or ODD performance. Using the annotated data probability distribution block 571, it is possible to determine the ODD coverage for an autonomous vehicle based on a metric. Using the annotated data probability distribution block 571, it is possible to determine the ODD performance for an autonomous vehicle based on a metric. That is, it is possible to evaluate the coverage and / or performance of a set of tests across the vehicle's stack target ODD (e.g., the stack of an autonomous vehicle) using a metric. It is possible to select one or more sampling algorithms to optimize parallelization to maximize subsequent metrics (e.g., metric, integrated metric, etc.) using a metric.

[0114] It may be possible to determine specific tests that may be used to facilitate the collection of data to maximize one or more of the performance or coverage of the integrated parameter space using metrics. In one example, the metric may indicate that light detection and ranging (LIDAR) may be used to increase one or more of the performance or coverage of the integrated parameter space. In another example, the user interface may receive an input that can select a specific test based on a specific use case. To facilitate data collection, a selected number of different algorithms and / or metrics associated with one or more tests may be provided. The user interface may be capable of receiving an input for selecting one or more of a specific test (such as LIDAR), a specific algorithm, a specific metric, etc.

[0115] It is possible to use metrics along with an internal parameter format to minimize data collection time and computational resources and maximize the quality of the collected data. A device for vehicle testing may include a memory and a processor operably coupled to the memory. The processor may be configured to execute instructions to cause the device to identify a first parameter set defined using a first scenario format, the first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to execute instructions to cause the device to map the first parameter set to an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to execute instructions to cause the device to calculate a metric based on one or more of a performance metric or a coverage metric. The device may include a display device configured to display the result based on the metric. This metric may be based on the use case type.

[0116] Using the annotated data probability distribution block 571, it is possible to provide metrics to a sampling service (such as the sampling service 450b shown in FIG. 4A, for example) and / or an inference service (such as the inference service 450c shown in FIG. 4A, for example). The sampling service 450b and / or the inference service 450c may be configured to perform one or more of: (i) select a set of axes based on the metrics, (ii) display results based on the set of axes on a display device, (iii) select a subset of the set of axes, or (iv) display a subset of the results based on the subset of the set of axes on a display device. The sampling service 450b and / or the inference service 450c may be configured to extract one or more axis values for one or more test modalities and calculate one or more axis-specific metrics based on the one or more axis values. For example, it is possible to send axis-specific metrics based on one or more axis values based on the metrics to the scenario cross-compiler 310 for transmission to the linker service 330 for generating test configuration objects.

[0117] Metrics can be used with asynchronous test sample results to minimize test time and computational resources and maximize one or more of subsequent test result coverage or performance. A device for vehicle simulation, testing, or verification may include a memory and a processor operably coupled to the memory. The processor may cause the device to perform one or more of: (i) testing a first test sample batch including one or more first test samples to output one or more first test sample results; (ii) identifying one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is complete; (iii) calculating a metric including one or more of a performance metric or a coverage metric based on the one or more asynchronous test sample results; or (iv) adjusting a second test sample batch based on the metric.

[0118] The processor may be configured to determine one or more asynchronous test sample results based on a parametric metric that may include one or more of a parametric performance metric or a parametric coverage metric. The asynchronous test sample results may be determined based on real-world data.

[0119] The first test sample results may be generated using one or more different test modalities (such as SIL, HIL, track, etc.). The first test samples may be converted from an internal parameter format to a test modality format. Asynchronous test sample results that may be selected from the first sample batch may be converted from the test modality format to the internal parameter format.

[0120] Using a metric, it may be possible to terminate the first sample batch before the first sample batch completes the test. Using a metric, it is possible to adjust a second test sample batch based on test adjustment configuration parameters including one or more of a selected sampling algorithm, a randomness parameter, or other hyperparameters such as a learning rate decay parameter or a sample size parameter.

[0121] As shown in FIG. 5, an annotated data probability distribution block 571 such as may be sent to an annotated data probability distribution block 572 specific to the test modality. The annotated data probability distribution block 572 may be configured to generate various test modality-specific distributions and sub-distributions including (i) an SIL-specific annotated distribution 573a, (ii) a HIL-specific annotated distribution 573b, (iii) a track-specific annotated distribution 573c, etc. These test modality-specific (e.g., 573a, 573b, 573c, etc.) annotated distributions can be sent to an extended test modality-specific data probability distribution block 574 for transmission to an intelligent job manager 540c.

[0122] The extended test modality-specific data probability distribution block 574 may include one or more of one or more distributions or sub-distributions such as SIL, HIL, drive data, etc. The extended test modality-specific data probability distribution block 574 may be extended by various tests performed using a conversion function 590 between two data probability distribution blocks by including factors related to increased variance and / or uncertainty resulting from the conversion.

[0123] The annotated data probability distribution block 571 may be configured to send the annotated data probability distribution to a learning correlation block 580 for processing. The output from the learning correlation block 580 may be sent to a conversion function 590 between two data probability distribution blocks that may be configured to output to the extended test modality-specific data probability distribution block 574.

[0124] Conversion functions between multiple test modalities Devices for vehicle testing, simulation, and verification may include a memory and a processor. The processor may be configured to execute instructions for causing the device to identify first test data based on a first test modality, where the test data includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data. For example, the learning correlation block 580 may be configured to receive data from the annotated data probability distribution block 571, as shown in FIG. 5. As shown in FIG. 6, the data provided from the probability distribution block 670 to the learning correlation block 680 may be one or more extended data point probability distributions (e.g., extended data point probability distribution 1, 681a, and extended data point probability distribution 2, 681b).

[0125] The conversion functions may be generated to switch between different test data modalities. The processor may be configured to execute instructions for causing the device to compute a conversion function configured to convert test data from a first test modality to a second test modality different from the first test modality. For example, the cross-distribution inference block 682 may be configured to generate conversion functions 686a, 686b that are (i) a conversion function from the extended data point probability distribution 1, 681a to the extended data point probability distribution 2, 681b, and (ii) a conversion function from the extended data point probability distribution 2, 681b to the extended data point probability distribution 1, 681a. The conversion functions 686a, 686b may be generated using any suitable function including one or more of (a) a non-parametric function 683 (e.g., a kernel), (b) a parametric function 684, or (c) a deep learning-based learning function 685.

[0126] Using a conversion function, it is possible to calculate test data of a test modality different from the received test data (such as the first test data of the first test modality) (such as the second test data of the second test modality). The processor may be configured to calculate second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data using the conversion function. The first test modality may be any suitable test modality including one or more of SIL, HIL, track test, etc. The second test modality may be any suitable test modality including one or more of SIL, HIL, track test, etc.

[0127] It is possible to calculate several metrics and compare the difference between the first test data and the second test data that may result as a result of generating the second test data using the first test data and the conversion function between the first test data and the second test data. These metrics may include (i) an information loss metric for determining the amount of information loss that occurs when the first test data is converted to the second test data using the conversion function, (ii) a risk metric for determining the amount of risk that occurs when the first test data is converted to the second test data using the conversion function, or (iii) an uncertainty metric for determining the amount of uncertainty that occurs when the first test data is converted to the second test data using the conversion function.

[0128] Using some of the metrics, it is possible to determine one or more reductions in test time or computational complexity that may occur when the conversion function is used compared to when it is not. The processor may be configured to calculate one or more of a conversion test time or a conversion computational complexity for calculating second test data using a conversion function based on first test data. The processor may be configured to calculate one or more of a non-conversion test time or a non-conversion computational complexity for calculating second test data without using the conversion function and the first test data. The processor may be configured to calculate one or more of a test time savings or a computational complexity savings based on a difference between (i) a conversion test time and a non-conversion test time, or (ii) a conversion computational complexity and a non-conversion computational complexity.

[0129] It is possible to classify using a conversion function to determine an ODD space. For example, when testing for different weather conditions, a sunny test scenario may be executed using sunny weather, and a rainy test scenario may be executed using rainy weather. The conversion function may be configured to convert one or more of a sunny test scenario or a rainy test scenario to collect data related to a test scenario for additional weather conditions (e.g., snowy, windy, dark, etc.).

[0130] The output of the learning correlation block 680 (e.g., the conversion function 690 between two data probability distributions) can be provided to the probability distribution block 570 (e.g., the annotated data probability distribution block 571) shown in FIG. 5 and transmitted to the data probability distribution 574 specific to the extended test modality. The output of the annotated data probability distribution specific to the test modality (e.g., 573a, 573b, 573c, etc.) may be provided to the data probability distribution 574 specific to the extended test modality. The data probability distribution 574 specific to the extended test modality may be transmitted to the intelligent job manager 540c.

[0131] FIG. 7 shows a process flow of an exemplary method 700 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 700 may be arranged according to at least one example described in the present disclosure.

[0132] Method 700 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in a processing device 1702 (such as a processor, for example) of FIG. 17, or another device, a combination of devices, or a system.

[0133] Method 700 may begin at block 705, where the processing logic can identify a first parameter set defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints.

[0134] At block 710, the processing logic can map the first parameter set to an internal parameter set using an internal parameter format that includes one or more internal axes, one or more internal relationships, and one or more internal constraints.

[0135] At block 715, the processing logic can generate a test configuration object configured to link to one or more objects using the internal parameter set.

[0136] At block 720, the processing logic can send the test configuration object to a test modality for testing.

[0137] The processor may further be configured to execute instructions to cause the device to identify a compile source of an internal parameter format for mapping the internal parameter set to a first parameter set, identify a metric of the internal parameter format after testing, format a test configuration object into a test modality format for a test modality, send the test configuration object to a test modality associated with the test modality, convert test modality format data into data in the internal parameter format in an internal parameter format converter plugin, or convert axis data of a time-based metric signal format into data in the internal parameter format in an internal parameter format converter plugin, or write the axis data to a database. The test modality may include one or more of a specific scenario language simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface. The first scenario format may include an open scenario format that may include an OSC definition language.

[0138] Modifications, additions, or omissions may be made to method 700 without departing from the scope of the present disclosure. For example, in some instances, method 700 may include any number of other components that may not be explicitly illustrated or described.

[0139] FIG. 8 shows a process flow of an exemplary method 800 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 800 may be arranged according to at least one example described in the present disclosure.

[0140] Method 800 may be executed by processing logic that may include hardware (such as circuitry, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0141] Method 800 may begin at block 805, and the processing logic may be capable of identifying a first set of parameters defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationship, and one or more first scenario format constraints.

[0142] At block 810, the processing logic may be capable of mapping the one or more internal axes and the first set of parameters to an internal set of parameters that includes one or more internal relationships and one or more internal constraints.

[0143] At block 815, the processing logic may be capable of calculating a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0144] The processor may further be configured to execute instructions for causing the device to select a set of axes, display a result based on the set of axes on a display device, or select a subset of the set of axes, display a subset of the result based on the subset of the set of axes on the display device, retrieve one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on the one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on a parametric metric, among one or more of these. The parametric metric may be based on a use case.

[0145] Modifications, additions, or omissions may be made to method 800 without departing from the scope of the present disclosure. For example, in some examples, method 800 may include any number of other components that may not be explicitly illustrated or described.

[0146] FIG. 9 shows a process flow of an exemplary method 900 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 900 may be arranged according to at least one example described in the present disclosure.

[0147] Method 900 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0148] Method 900 may be capable of starting at block 905 where the processing logic is capable of receiving scenario format data.

[0149] In block 910, the processing logic can convert the scenario format data into data in the internal parameter format.

[0150] In block 915, the processing logic can generate a test configuration object using the data in the internal parameter format.

[0151] In block 920, the processing logic can test the test configuration object using the test modality.

[0152] The processor further configures the device to execute instructions for performing one or more of: formatting the test configuration object into a test modality format for the test modality, sending the test configuration object to a test modality associated with one or more of a specific scenario format simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface, converting the test modality format data into data in the internal parameter format in an internal parameter format converter plugin, converting the axis data of the time-based metric signal format into data in the internal parameter format in an internal parameter format converter plugin, writing the axis data to a database, calculating a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric and based on a use case, extracting one or more axis values for the test modality, calculating one or more axis-specific metrics based on the one or more axis values, and generating the test configuration object based on one or more axis-specific metrics based on the one or more axis values selected based on the parametric metric.

[0153] Modifications, additions, or omissions may be made to method 900 without departing from the scope of the present disclosure. For example, in some instances, method 900 may include any number of other components that may not be explicitly illustrated or described.

[0154] FIG. 10 shows a process flow of an exemplary method 1000 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1000 may be arranged according to at least one example described in the present disclosure.

[0155] Method 1000 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0156] Method 1000 may begin at block 1005, and the processing logic may be capable of testing a first test sample batch that includes one or more first test samples and outputting one or more first test sample results.

[0157] At block 1010, the processing logic may be capable of identifying one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed.

[0158] At block 1015, the processing logic may be capable of calculating a parametric metric that includes one or more of a parametric performance metric or a parametric coverage metric based on one or more asynchronous test sample results.

[0159] In block 1020, the processing logic can adjust the second test sample batch based on parametric metrics.

[0160] The processor causes the device to determine one or more asynchronous test sample results based on real-world data, determine one or more asynchronous test sample results based on a parametric metric that includes one or more of a parametric performance metric or a parametric coverage metric, end the first sample batch based on the parametric metric, convert a plurality of first test samples from an internal parameter format to a test modality format, convert one or more asynchronous test sample results from a test modality format to an internal parameter format, and may be further configured to execute instructions to cause the device to adjust a second test sample batch based on one or more of test adjustment configuration parameters that include one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, etc. The plurality of first test sample results may be generated using one or more different test modalities. Alternatively, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (such as a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0161] Modifications, additions, or omissions may be made to method 1000 without departing from the scope of the present disclosure. For example, in some examples, method 1000 may include any number of other components that may not be explicitly illustrated or described.

[0162] FIG. 11 shows a process flow of an exemplary method 1100 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1100 may be arranged according to at least one example described in the present disclosure.

[0163] Method 1100 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0164] Method 1100 may start at block 1105, and the processing logic may be capable of identifying one or more asynchronous test sample results of one or more first test samples before the first test sample batch completes the test.

[0165] At block 1110, the processing logic may be capable of calculating a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0166] At block 1115, the processing logic may be capable of selecting a set of algorithms based on the parametric metric.

[0167] At block 1120, the processing logic may apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch completes the test.

[0168] The processor may further be configured to execute instructions to cause the device to calculate a parametric metric including one or more of a confidence level, a confidence interval, a normalized metric, or a sample density, select a set of axes based on the parametric metric, display a result based on the set of axes on a display device, select a subset of the set of axes, display a subset of the result based on the subset of the set of axes on the display device, extract one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on the one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric. The parametric metric may be based on a use case. The parametric metric may be calculated using one or more of an objective function or real-world data. Alternatively, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (such as a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0169] Modifications, additions, or omissions may be made to method 1100 without departing from the scope of the present disclosure. For example, in some instances, method 1100 may include any number of other components that may not be explicitly illustrated or described.

[0170] FIG. 12 shows a process flow of an exemplary method 1200 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1200 may be arranged according to at least one example described in the present disclosure.

[0171] Method 1200 may be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0172] Method 1200 may start at block 1205, and the processing logic may test a first test sample batch in a test modality format, where the first sample batch includes one or more first test samples for outputting one or more first test sample results.

[0173] At block 1210, the processing logic can identify one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed.

[0174] At block 1215, the processing logic can convert one or more asynchronous test sample results from the test modality format to an internal parameter format.

[0175] At block 1220, the processing logic can adjust a second test sample batch based on one or more asynchronous test sample results.

[0176] The processor causes the device to format a second test sample batch into a test modality format for test modality, or send the second test sample batch to a test modality associated with the test modality that includes one or more of a specific scenario format simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface, or convert axis data of a time-based metric signal format into data in an internal parameter format in an internal parameter format converter plugin, or write the axis data to a database, or calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric and based on a use case, or extract one or more axis values for a test modality, or calculate one or more axis-specific metrics based on one or more axes, and generate a second test sample batch based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric, and may be further configured to execute instructions for causing one or more of the above to be performed. Alternatively, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (such as a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0177] Modifications, additions, or omissions may be made to method 1200 without departing from the scope of the present disclosure. For example, in some instances, method 1200 may include any number of other components that may not be explicitly illustrated or described.

[0178] FIG. 13 shows a process flow of an exemplary method 1300 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1300 may be arranged according to at least one example described in the present disclosure.

[0179] Method 1300 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0180] Method 1300 may start at block 1305, and the processing logic is capable of calculating an internal parameter set that includes one or more internal axes, one or more internal relationships, and one or more internal constraints.

[0181] At block 1310, the processing logic is capable of calculating a metric that includes one or more of a performance metric or a coverage metric using the internal parameter set.

[0182] The processor may further be configured to execute instructions to cause the device to calculate a metric for the use case, calculate a metric using one or more of a confidence level, a confidence interval, a normalized format, or a sample density, determine an operational design domain (ODD) coverage of the vehicle based on the metric, determine an ODD performance of the vehicle based on the metric, query a safety characteristic of the vehicle, calculate a safety metric based on the metric in response to querying the safety characteristic of the vehicle, calculate an integrated metric based on a performance metric and a coverage metric, calculate an integrated metric based on one or more of an objective function or real-world data, or calculate a metric using parallelization to maximize one or more of the ODD coverage or the ODD performance.

[0183] Modifications, additions, or omissions may be made to method 1300 without departing from the scope of the present disclosure. For example, in some instances, method 1300 may include any number of other components that may not be explicitly illustrated or described.

[0184] FIG. 14 shows a process flow of an exemplary method 1400 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1400 may be arranged according to at least one example described in the present disclosure.

[0185] Method 1400 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0186] Method 1400 may start at block 1405, and the processing logic can identify a first set of parameters defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints.

[0187] At block 1410, the processing logic can map the first set of parameters to an internal set of parameters that includes one or more internal axes, one or more internal relationships, and one or more internal constraints.

[0188] At block 1415, the processing logic can calculate a metric based on one or more of a performance metric or a coverage metric.

[0189] The processor may be further configured to execute instructions for causing the device to perform one or more of: select a set of axes based on the metric, display a result based on the set of axes on a display device, select a subset of the set of axes, display a subset of the result based on the subset of the set of axes on a display device, retrieve one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on the one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the metric. The metric may be based on a use case.

[0190] Modifications, additions, or omissions may be made to method 1400 without departing from the scope of the present disclosure. For example, in some examples, method 1400 may include any number of other components that may not be explicitly illustrated or described.

[0191] FIG. 15 shows a process flow of an exemplary method 1500 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1500 may be arranged according to at least one example described in the present disclosure.

[0192] Method 1500 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0193] Method 1500 may begin at block 1505, and the processing logic may be capable of testing a first test sample batch that includes one or more first test samples and outputting one or more first test sample results.

[0194] At block 1510, the processing logic may be capable of identifying one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed.

[0195] At block 1515, the processing logic may be capable of calculating a metric that includes one or more of a performance metric or a coverage metric based on the one or more asynchronous test sample results.

[0196] At block 1520, the processing logic may be capable of adjusting a second test sample batch based on the metric.

[0197] The processor causes the device to determine one or more asynchronous test sample results based on parametric metrics including one or more of parametric performance metrics or parametric coverage metrics, or to determine one or more asynchronous test sample results based on real-world data, or to end a first sample batch based on a metric, or to convert a plurality of first test samples from an internal parameter format to a test modality format, or to convert one or more asynchronous test sample results from a test modality format to an internal parameter format, or to further configure to execute instructions to adjust a second test sample batch based on test adjustment configuration parameters including one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, and the like. The first test sample results may be generated using one or more different test modalities.

[0198] Modifications, additions, or omissions may be made to method 1500 without departing from the scope of the present disclosure. For example, in some examples, method 1500 may include any number of other components that may not be explicitly illustrated or described.

[0199] FIG. 16 shows a process flow of an exemplary method 1600 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1600 may be arranged according to at least one example described in the present disclosure.

[0200] Method 1600 may be executed by processing logic that may include hardware (such as a circuit, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0201] Method 1600 may begin at block 1605, and the processing logic can identify first test data based on a first test modality that includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data.

[0202] At block 1610, the processing logic can calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality.

[0203] At block 1615, the processing logic can use the conversion function to calculate second test data that includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data.

[0204] The processor causes the device to calculate an information loss metric, which is the amount of information loss for the second test data compared to the first test data, based on one or more of the first test data or the second test data; or calculate a risk metric, which is the amount of risk for the second test data compared to the risk for the first test data, based on one or more of the first test data or the second test data; or calculate an uncertainty metric, which is the amount of uncertainty for the second test data compared to the uncertainty for the first test data, based on one or more of the first test data or the second test data; or calculate a conversion test time for calculating the second test data based on the first test data using a conversion function; or calculate a non-conversion test time for calculating the second test data without using the conversion function and the first test data; or calculate a test time savings based on the difference between the conversion test time and the non-conversion test time, and may be further configured to execute an instruction for causing the device to perform one or more of the above. The first test modality may be one or more of software-in-the-loop (SIL), hardware-in-the-loop (HIL), and track test. The second test modality may be one or more of software-in-the-loop (SIL), hardware-in-the-loop (HIL), and track test.

[0205] Modifications, additions, or omissions may be made to method 1600 without departing from the scope of the present disclosure. For example, in some examples, method 1600 may include any number of other components that may not be explicitly illustrated or described.

[0206] To simplify the description, the methods and / or process flows described herein are depicted and described as a series of acts. However, the acts according to the present disclosure can be performed in various orders and / or simultaneously, and together with other acts not presented and described herein. Further, not all of the acts shown may be used to implement the methods according to the disclosed subject matter. Additionally, those skilled in the art will understand and recognize that the methods may alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, the methods disclosed herein can be stored on a manufactured article such as a non-transitory computer-readable medium to facilitate transmission and transfer of such methods to a computing device. As used herein, the term manufactured article is intended to encompass a computer program accessible from any computer-readable device or storage medium. Although shown as individual blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

[0207] In some examples, a vehicle simulation, test, and verification system (e.g., for AVs, etc.) may include an environmental representation creation module with code and routines configured to enable a computing device to perform one or more operations such as generating a 3D environmental representation, preparing and generating scenarios, simulating scenarios, and verifying and / or evaluating the simulation of scenarios. Additionally, or alternatively, the environmental representation creation module may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling the performance of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other instances, the environmental representation creation module may be implemented using a combination of hardware and software. In the present disclosure, the operations described as being performed by the environmental representation creation module may include operations that can be instructed to be performed by the system corresponding to the environmental representation creation module.

[0208] In some examples, the environmental representation creation module may be configured to generate a 3D environmental representation. The environmental representation creation module can generate a 3D environmental representation by any suitable 3D modeling technique. In some examples, the environmental representation creation module may use map data as input data in generating the 3D environmental representation. For example, the 3D environment of the 3D environmental representation may represent a geographical area represented by the map data.

[0209] In some examples, the 3D environmental representation may include 3D models of one or more objects in a geographical area as described by the map data. For example, the 3D environmental representation may include a complete 3D model of a simulated driving environment.

[0210] A vehicle simulation, test, and verification system (e.g., for AV, etc.) may include a machine learning circuit and / or software for more efficiently and preventively identifying bad cases. A vehicle simulation, test, and verification system (e.g., for AV, etc.) may include a circuit and / or software for pruning, including parameter-based pruning. A vehicle simulation, test, and verification system (e.g., for AV, etc.) may include a circuit and / or software for implementing scenarios, runtime constraints for simulations, or runtime constraints within a 3D environment representation.

[0211] FIG. 17 shows a diagrammatic representation of a machine in an exemplary form of a computing device 1700 on which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein. Computing device 1700 may include, for example, a rack-mounted server, a router computer, a server computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, or any computing device having at least one processor, within which a set of instructions for causing the machine to perform any one or more of the methods discussed herein may be executed. In an alternative example, the machine may be connected (e.g., network-connected) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine may be capable of operating in the capacity of a server machine in a client-server network environment. Further, although only a single machine is shown, the term "machine" may also include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions for performing any one or more of the methods discussed herein.

[0212] The exemplary computing device 1700 includes a processing device 1702 (such as a processor), a main memory 1704 (such as dynamic random access memory (DRAM) including read-only memory (ROM), flash memory, synchronous DRAM (SDRAM), etc.), a static memory 1706 (such as flash memory, static random access memory (SRAM), etc.), and a data storage device 1716, and these communicate with each other via a bus 1708.

[0213] The processing device 1702 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. More specifically, the processing device 1702 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set, or a processor implementing a combination of instruction sets. The processing device 1702 may also include one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device 1702 is configured to execute instructions 1726 for performing the operations and steps discussed herein.

[0214] Computing device 1700 may further include a network interface device 1722 capable of communicating with network 1718. Computing device 1700 may also include a display device 1710 (such as a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1712 (such as a keyboard), a cursor control device 1714 (such as a mouse), and a signal generation device 1720 (such as a speaker). In at least one example, the display device 1710, the alphanumeric input device 1712, and the cursor control device 1714 may be combined into a single component or device (such as an LCD touch screen).

[0215] Data storage device 1716 may include a computer-readable storage medium 1724 storing one or more sets of instructions 1726 embodying any one or more of the methods or functions described herein. The instructions 1726 may also be present, in whole or at least in part, within main memory 1704 and / or within processing device 1702 during execution by computing device 1700, and main memory 1704 and processing device 1702 also constitute computer-readable media. The instructions may also be further transmitted or received via network interface device 1722 over network 1718.

[0216] The computer-readable storage medium 1724 is shown as a single medium in one example, but the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) that store a set of one or more instructions. The term "computer-readable storage medium" may also include any medium that can store, encode, or carry a set of instructions for machine execution and cause a machine to execute any one or more of the methods of the present disclosure. Thus, the term "computer-readable storage medium" may include, but is not limited to, solid-state memory, optical media, and magnetic media.

Example

[0217] The following provides examples of the present disclosure.

[0218] Example 1: Lane Support Using an Open Scenario In this example, the target vehicle and the host vehicle are described with respect to the lane. The host vehicle travels along a route and is initially in the same lane as the target vehicle. The target vehicle is accelerating towards the host vehicle and has a distance of approximately 0.1 meter to the right lane dividing line. At the start of the scenario, the host vehicle has a distance of approximately 100 m between itself and the target vehicle and a distance of approximately 0.5 to the right lane dividing line. During the scenario, the host vehicle moves to a different lane. The host vehicle returns to the same lane as the target vehicle at the end of the scenario.

[0219] An example of the OpenSCENARIO 2.0 scenario format is provided in Table I.

Table 1

[0220] Example 2: Conversion from OpenSCENARIO to an Internal Parameter Format The OpenSCENARIO of the scenario description language may be converted into an internal parameter format.

[0221] The OpenSCENARIO code provided in Table I includes (i) axes (such as, for example, vehicle, route, lane, lateral, position, speed, etc.), (ii) relationships (such as, for example, target vehicle and host vehicle in the same lane, etc.), and (iii) constraints (such as, for example, at least two lanes, etc.). The OpenSCENARIO code may be mapped to an internal parameter format by identifying the OpenSCENARIO axes, relationships, and constraints and converting those axes, relationships, and constraints into an internal parameter format (such as, for example, an intermediate YAML format, etc.).

[0222] Example 3: Parallel Tests Using Asynchronous Information Two different scenarios of the OpenSCENARIO language may be converted into an internal parameter format. Both scenarios may be tested in the same batch as the first test and the second test. The first test based on the first scenario may complete the test before the second test based on the scenario completes the test. The result of the first test based on the first scenario may be used in the third test based on the second scenario.

[0223] Provide the first scenario in Table I and the second scenario in Table II. [Table 2]

[0224] In the second scenario, the host vehicle does not change lanes in response to the target vehicle accelerating behind the host vehicle in the same lane. The host vehicle accelerates from 100 km / h to 130 km / h. The target vehicle starts the scenario at a speed of 150 km / h. It is possible to use this scenario to test the response of the host vehicle to an approaching vehicle.

[0225] The first test results from the first scenario may indicate that if the host vehicle does not change lanes, the host vehicle may collide with the target vehicle from behind. These first test results may be used to adjust the third test based on the second scenario if completed before the completion of the second test results. Using the knowledge that the target vehicle may collide with the host vehicle without the host vehicle being able to change lanes, for example, by reducing the speed of the target vehicle and testing whether the host vehicle can sense the target vehicle and avoid the collision, it is possible to modify the third scenario.

[0226] Example 4: Parametric Metric Using Internal Parameter Format and Asynchronous Information It is possible to calculate parametric metrics using the scenarios detailed in Table I ("following") and II ("preceding"). Converting the scenarios into an internal parameter format (such as an intermediate YAML format) can enhance the performance and information gain received for parametric metrics (such as one or more of parametric performance metrics or parametric coverage metrics).

[0227] Convert the scenarios represented using OpenSCENARIO into an internal parameter format to facilitate conversion to different test modalities (such as SIL, HIL, track, etc.) of the scenarios and to convert the test results back from different test modalities to the internal parameter format. Metrics from different test modalities may be compared using the same language. Therefore, the parametric coverage metric and the parametric performance metric may be calculated with higher accuracy and precision compared to calculations and comparisons based on different languages.

[0228] For the scenarios described in the trailing and leading scenarios, one performance metric may be the frequency with which the host vehicle collides as a result of: (i) changing lanes so that the target vehicle can pass, or (ii) staying in the same lane and increasing speed. The coverage metric may be calculated based on the number of different (e.g., different weather, vehicle speed, road type, etc.) situations associated with these scenarios in the trailing and leading scenarios that are tested based on test data from the trailing and leading scenarios.

[0229] Converting the OpenSCENARIO format to an internal parameter format (such as, for example, an intermediate YAML format) makes it possible to combine data from the OpenSCENARIO format with data from a different scenario language (i.e., not the OpenSCENARIO language) and to calculate metrics based on the combined data. Different scenario languages can collect data related to weather conditions and road types for the host vehicle and the target vehicle in different scenarios (such as, for example, a composite lane change scenario where the host vehicle and the target vehicle switch lanes almost simultaneously while traveling within a selected distance from each other). The data from this scenario may be converted to an internal parameter format (such as, for example, an intermediate YAML format) and combined with data from the scenarios described in the trailing and leading scenarios. The sum of the coverage metric as provided by the trailing and leading and the coverage metric as provided by the composite lane change scenario may not be the same as the coverage metric when the scenarios described in Tables I and II are combined with the coverage metric before calculating the coverage metric.

[0230] It is possible to increase information gain and reduce computational complexity and test time using asynchronous test results. For situations where the trailing scenario and the composite lane change scenario complete the test before the leading scenario completes the test, the trailing scenario and the composite lane change scenario may be converted to an internal parameter format (such as an intermediate YAML format, etc.) before the performance and / or coverage metric is calculated. Using the resulting performance and / or coverage metric, it is possible to select a specific algorithm to be applied to subsequent test batches before the test data from the leading scenario is complete.

[0231] Example 5: Metrics Using Internal Parameter Format and Asynchronous Information Trailing, leading, and composite lane change scenarios may be used with the metric. This metric may be calculated using one or more of a confidence level, a confidence interval, a normalized format, or a sample density. For example, the trailing scenario may have a collision rate of 2%, the leading scenario may have a collision rate of 3%, and the composite lane change scenario may have a collision rate of 5%. The confidence level and interval may be calculated for these scenarios. The 2% collision rate of the trailing scenario may have a confidence interval of 2.5% - 3.5% with a confidence level of 95%. The 3% collision rate of the leading scenario may have a confidence interval of 3.5% - 4.5% with a confidence level of 97%. The 5% collision rate of the composite lane change scenario may have a confidence interval of 3% - 8% with a confidence level of 85%.

[0232] The normalized format may be calculated by combining three different scenarios using the internal parameter format and normalizing the data such that the area under the curve is approximately equal to 1. The sample density may be calculated based on the sample density for each of the scenarios.

[0233] The metric may be combined into a single metric that combines a coverage metric and a performance metric (e.g., an integrated metric, etc.). It is possible to determine what to test using a single metric. For example, a coverage metric based on trailing, leading, and combined lane change scenarios may indicate that in a blizzard, there is no coverage for a scenario where two vehicles simultaneously change lanes to the same lane and travel at a speed of 300 km / h, but the single metric can calculate that the scenario will not be tested because the performance (e.g., collision, etc.) is known.

[0234] Example 6: Conversion Function The first test data calculated using the first test modality may be converted into second test data for the second test modality. For example, the first test data may be SIL test data, and the second test data may be HIL data. A suitable machine learning technique such as supervised machine learning (e.g., regression, etc.) using training data for the first test modality and the second test modality may be used to determine the conversion function between the first test modality and the second test modality. The objective function may be optimized to generate one or more functions used to convert between multiple test modalities.

[0235] For trailing, leading, and combined lane change scenarios, the test modality may include SIL test data. The SIL test data may be converted into HIL test data using the conversion function. Converting the SIL test data into HIL test data using the conversion function can facilitate shortening the test time.

[0236] In some examples, the different components, modules, engines, and services described herein may be implemented as objects or processes (e.g., as separate threads) running on a computing system. Some of the systems and methods described herein are generally described as being implemented in software (stored in hardware and / or executed by hardware), but specific hardware implementations, or combinations of software and specific hardware implementations, are also possible and contemplated.

[0237] As used herein, terms, particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be “open” terms (e.g., the term “comprising” should be interpreted as “comprising but not limited to,” the term “having” should be interpreted as “having at least,” the term “including” should be interpreted as “including but not limited to,” and so forth).

[0238] In addition, where a specific number of introduced claim recitations is intended, such intent is to be explicitly recited in the claim, and where there is no such recitation, there is no such intent. For example, by way of illustration, the following appended claims may include the use of the preambles “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim that includes such introduced claim recitation to examples that include only one such recitation, even where the same claim includes a preamble such as “one or more” or “at least one” and an indefinite article such as “a” or “an” (e.g., “a” and / or “an” should be interpreted to mean “at least one” or “one or more”). The same holds true for the use of definite articles used to introduce claim recitations.

[0239] In addition, it is understood that even if the specific number of the recited items introduced is explicitly recited, such recitation should be construed to mean at least the recited number (for example, a literal recitation of "two recited items" without other modifiers means at least two recited items, or two or more recited items). Further, in those cases where conventions similar to "at least one of A, B, and C" or "one or more of A, B, and C" are used, generally such a construction is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc. For example, the use of the term "and / or" is intended to be construed in this way.

[0240] Furthermore, regardless of the description, claims, or drawings, any disjunctive word or phrase presenting two or more alternative terms should be understood to contemplate the possibility of including one of the terms, any of the terms, or both terms. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0241] In addition, the use of terms such as "first", "second", "third", etc. is not necessarily used herein to mean a particular order or number of elements. In general, terms such as "first", "second", "third", etc. are used as general identifiers to distinguish different elements from each other. When it is not indicated that terms such as "first", "second", "third", etc. mean a particular order, these terms should not be understood to mean a particular order. Further, when it is not indicated that terms such as "first", "second", "third", etc. mean a particular number of elements, these terms should not be understood to mean a particular number of elements. For example, a first so-and-so device may be described as having a first surface, and a second so-and-so device may be described as having a second surface. The use of the term "second surface" with respect to the second so-and-so device may be to distinguish such a surface of the second so-and-so device from the "first surface" of the first so-and-so device, and does not mean that the second so-and-so device has two surfaces.

[0242] All examples and conditional language recited herein are intended for the educational purpose of helping the reader understand the concepts presented by the inventors to promote the present invention and the technology, and should be construed as not being limited to such specifically recited examples and conditions. Although the examples of the present disclosure have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure.

Claims

1. A device for vehicle testing, the device comprising: a memory; a processor operably coupled to the memory, the processor executing instructions to cause the device to: test a first test sample batch including a plurality of first test samples to output a plurality of first test sample results; prior to completion of the first test sample batch, identify one or more asynchronous test sample results among the plurality of first test samples; calculate a parametric metric including one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results; adjust a second test sample batch based on the parametric metric; a processor configured as such,

2. The processor further executes instructions to cause the device to: determine the one or more asynchronous test sample results based on a parametric metric including one or more of a parametric performance metric or a parametric coverage metric, according to the device of Claim 1 configured as such.

3. The processor further executes instructions to cause the device to: determine the one or more asynchronous test sample results based on real-world data, according to the device of Claim 1 configured as such.

4. The processor further executes instructions to cause the device to: end the first test sample batch based on the parametric metric, according to the device of Claim 1 configured as such.

5. The processor further executes instructions to cause the device to: convert the plurality of first test samples from an internal parameter format to a test modality format; convert the one or more asynchronous test sample results from the test modality format to the internal parameter format, according to the device of Claim 1 configured as such.

6. The processor further executes instructions to cause the device to: The device according to claim 1, configured to adjust the second test sample batch based on test adjustment configuration parameters including one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, and an algorithm selection parameter.

7. The device according to claim 1, wherein the plurality of first test sample results are generated using a plurality of different test modalities.

8. A device for vehicle testing, the device comprising: a memory; a processor operably coupled to the memory, the processor executing instructions to cause the device to: identify one or more asynchronous test sample results of a plurality of first test samples before the first test sample batch completes testing; calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric; select a set of algorithms based on the parametric metric; apply an algorithm from the set of algorithms to a second test sample batch before the first test sample batch completes testing, the processor being configured to: device.

9. The device according to claim 8, wherein the parametric metric is based on a use case.

10. The device according to claim 8, wherein the parametric metric is calculated using one or more of an objective function or real-world data.

11. The processor is further configured to execute instructions to cause the device to: calculate a parametric metric including one or more of a confidence level, a confidence interval, a normalized metric, or a sample density, the device according to claim 8.

12. The processor is further configured to execute instructions to cause the device to: select a set of axes based on the parametric metric, or display a result based on the set of axes on a display device, or select a subset of the set of axes, or display a subset of the results based on the subset of the set of axes on the display device, the device according to claim 8.

13. The processor is further configured to execute instructions to cause the device to extract one or more axis values for one or more test modalities, and calculate one or more axis-specific metrics based on the one or more axis values, the device according to claim 8. **Claim 14** The processor is further configured to execute instructions to cause the device to generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric, the device according to claim 8. **Claim 15** A computer-readable storage medium comprising instructions executable by a computer, the instructions, when executed by one or more processors, cause a vehicle tester to test a first test sample batch in a test modality format and including a plurality of first test samples, and output a plurality of first test sample results, prior to completion of the first test sample batch, identify one or more asynchronous test sample results among the plurality of first test samples, convert the one or more asynchronous test sample results from the test modality format to an internal parameter format, adjust a second test sample batch based on the one or more asynchronous test sample results, a computer-readable storage medium. **Claim 16** The instructions, when executed by the one or more processors, further cause the vehicle tester to format the second test sample batch in the test modality format for the test modality, send the second test sample batch to the test modality, wherein the test modality includes one or more of a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface, the computer-readable storage medium according to claim 15. **Claim 17** The instructions, when executed by the one or more processors, further cause the vehicle tester to In the internal parameter format converter plugin, the axis data in the time-based metric signal format is converted into the data in the internal parameter format. The computer-readable storage medium according to claim 15, which causes the axis data to be written into a database.

18. When the instructions are executed by the one or more processors, the vehicle tester is further caused to calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric and based on a use case. The computer-readable storage medium according to claim 15.

19. When the instructions are executed by the one or more processors, the vehicle tester is further caused to extract one or more axis values for a test modality, calculate one or more axis-specific metrics based on the one or more axis values. The computer-readable storage medium according to claim 15.

20. When the instructions are executed by the one or more processors, the vehicle tester is further caused to generate the second test sample batch based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric. The computer-readable storage medium according to claim 15.

Images