Scenario-independent internal representation of the parameter space

Abstract

The present disclosure relates to a device for vehicle simulation, testing, and verification. The device may comprise a memory and a processor operably coupled to the memory. The processor may be configured to identify a first set of parameters defined using a first scenario format, the first set of parameters including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first set of parameters to an internal set of parameters using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object using the internal set of parameters and transmit the test configuration object to a test modality for testing.

Description

Technical Field

[0001] (Related Applications) This application claims the benefit of U.S. Provisional Application No. 63 / 366,770, filed on June 21, 2022, and U.S. Provisional Application No. 63 / 366,830, filed on June 22, 2022, the disclosures of which are incorporated herein by reference in their entirety.

[0002] This disclosure relates to vehicle simulation, test, and verification systems and methods.

Background Art

[0003] Unless otherwise indicated herein, the subject matter described herein is not prior art to the claims in this application and is not admitted to be prior art by virtue of being included in this section.

[0004] Software can be tested using a simulated environment. For example, a simulated driving environment can be used to test the software of an autonomous vehicle. An autonomous vehicle can use sensors to perceive its environment. A control system can model the perception input from the sensors to determine a navigation route and make decisions in response to traffic controls such as stop signals, roundabouts, stop signs, speed limit changes, and other vehicles.

[0005] Vehicles can have various levels of automation. For some vehicles, the automation system may provide warnings but may not otherwise control the vehicle. For other vehicles, the vehicle can operate on many different surfaces, seasons, and weather conditions without human intervention. As a result, devices, systems, and methods for testing, simulating, and verifying vehicles with a wide range of automation levels can be useful.

[0006] The subject matter claimed in this disclosure is not limited to examples that solve any disadvantages or that function only in an environment, such as those described above. Rather, this background is provided only to illustrate one exemplary technical field in which some examples described in this disclosure may be implemented. SUMMARY OF THE INVENTION

[0007] Devices for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first set of parameters defined using a first scenario format, the first set of parameters including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first set of parameters to an internal set of parameters using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object configured to link to one or more objects using the internal set of parameters. The processor may be configured to transmit the test configuration object to a test modality for testing.

[0008] Devices for vehicle simulation, testing, and validation may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first set of parameters defined using a first scenario format, the first set of parameters including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first set of parameters to an internal set of parameters including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0009] A non-transitory computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, cause a vehicle simulator to receive scenario format data, convert the scenario format data to internal parameter format data, generate a test configuration object using the internal parameter format data, and test the test configuration object using a test modality.

[0010] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of one or more of the plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a parametric metric including one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second test sample batch based on the parametric metric.

[0011] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify one or more asynchronous test sample results of one or more of the plurality of first test samples before the first test sample batch completes testing. The processor may be configured to calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric. The processor may be configured to select a set of algorithms based on the parametric metric. The processor may be configured to apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch completes testing.

[0012] A computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, cause a vehicle tester to test a first test sample batch in a test modality format including one or more first test samples for outputting one or more first test sample results, identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed, convert the one or more asynchronous test sample results from the test modality format to an internal parameter format, and adjust a second test sample batch based on the one or more asynchronous test sample results.

[0013] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric using the internal parameter set.

[0014] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set defined using a first scenario format, the first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first parameter set to an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric based on one or more of a performance metric or a coverage metric.

[0015] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of a plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second test sample batch based on the metric.

[0016] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify first test data including one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data based on a first test modality. The processor may be configured to calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality. The processor may be configured to calculate second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data using the conversion function.

[0017] At least the elements, features, and combinations that are particularly noted in the claims achieve the example purposes and advantages.

[0018] Both the above summary and the following detailed description of the invention are given by way of example and are illustrative and not limiting of the claimed invention.

Brief Description of the Drawings

[0019] Examples will be described and explained with additional particularity and detail through the use of the accompanying drawings.

[0020]

Figure 1A

Figure 1B

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

[0021] A vehicle software test bed, which can be an autonomous vehicle software test bed, can be used to test vehicle software (or autonomous vehicle software). The vehicle software test bed can simulate an operating environment for testing the vehicle software (e.g., for determining how the vehicle software performs in various driving situations). For example, the vehicle software test bed can simulate a three-dimensional (3D) geographical environment and provide inputs to the vehicle software that are similar to the inputs that the vehicle software could receive under real-world driving conditions, assuming the vehicle software is operating on a vehicle while navigating in the actual geographical environment.

[0022] Determining a set of scenarios to generate and execute in a simulation is useful for efficiently testing and validating vehicle systems, and more specifically, autonomous vehicle (AV) systems. Identifying a set of scenarios that sufficiently cover the test space to a statistical confidence level helps demonstrate the accuracy and safety of the underlying system under test.

[0023] Previous approaches to scenario generation and coverage calculation are based on the following three underlying assumptions: (1) a scenario definition language optimized for easy (but not necessarily comprehensive) parameterization, (2) contextualization and introspection into the simulation environment and stack, and (3) access to a closed-loop test environment. From these three assumptions, a verification engineer can define the parameters that make up the verification engineer's tests, the bad conditions for the tests, and the range of possible values and step sizes for each parameter. Combinations of parameters are then repeatedly tested, and new combinations are selected to gather additional information about the stack. Next, the coverage calculation is based on which parameter values were tested across the entire set of tests. Coverage is calculated naively by binning the parameters into multi-dimensional histograms and clusters.

[0024] The limitations of these conventional approaches are that they rely on the details of each of the three assumptions above. In particular, much previous work has relied on a proprietary scenario definition format, which may limit the expressiveness and extensibility of many aspects of the scenario compared to using an industry-standard scenario language. Additionally, these scenario languages are often closely tied to the ability to introspect and modify these scenarios. Also, their closed-loop simulation environments for execution may not be realistic enough to provide accurate and realistic previous information for adaptive sampling. In addition to this, many of these algorithms rely entirely on previous samples collected before selecting the next sample to execute. Therefore, the ability of previous approaches for scaling is limited, and these approaches are mainly limited to the research context. Finally, naive approaches to computing coverage result in limitations in the ability to evaluate stack performance and test coverage in relation to how important different scenarios are.

[0025] Aspects of the present disclosure provide, among other things, systems and methods for enhanced simulation, scenario selection, and environment generation. For example, the systems and methods provide enhanced efficiency, such as through information sharing between different test modalities. The improved information sharing enables sampling to be done more logically and efficiently.

[0026] Additional efficiency gains can result from stack evaluation (e.g., performance and coverage), which could otherwise be a manual process, while a statistical interface can speed up the evaluation. Parallelizable simulation and performance improvements resulting from different test modalities can generate sampling algorithms that can converge faster. Also, using the disclosed techniques, AV tests can identify bad modes faster and statistically verify autonomous stacks.

[0027] Another aspect may provide a set of meta-statistics that quantify the amount of information loss that occurs based on this transformation when converting between different test modalities and risky / uncertain regions. The set of meta-statistics may quantify the amount of tests stored in each modality based on this correlation.

[0028] In yet another aspect, the present disclosure includes a method for correlating information from software-in-the-loop (SIL), hardware-in-the-loop (HIL), and track tests in order to make predictions regarding performance in different situations. The aspect may provide a definition of a conversion function between SIL, HIL, and track test performance, coverage, or metric distributions and the corresponding uncertainties.

[0029] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify a first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints defined using a first scenario format. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object configured to link to one or more objects using the internal parameter set. The processor may be configured to transmit the test configuration object to a test modality for testing.

[0030] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples and output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed. The processor may be configured to calculate a parametric metric including one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results. The processor may be configured to adjust a second test sample batch based on the parametric metric.

[0031] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric including one or more of a performance metric or a coverage metric using the internal parameter set.

[0032] Devices for vehicle simulation, testing, and verification may include a memory and a processor operably coupled to the memory. The processor may be configured to identify first test data including one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data based on a first test modality. The processor may be configured to calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality. The processor may be configured to use the conversion function to calculate second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data.

[0033] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0034] A scenario-independent internal representation of the parameter space can be used to map between open and / or custom representations and the internal representation. The parameter space itself can be manipulated to reduce the complexity and difficulty of problems for search / optimization algorithms. Some exemplary data representations may include a scenario definition language (Automation Systems and Measuring Systems Standardization Society (ASAM) Open Scenario (registered trademark) (OSC) 1.0, OSC 2.0, or any other scenario language), axes (variables and possible values of the variables, as well as output test metrics), relationships (defining how variables are related to each other), constraints (defining boundaries of the axes and limitations in the relationships), and the like.

[0035] As shown in FIG. 1A, a block diagram 100a for vehicle testing, simulation, and verification is provided. A set of parameters can be defined that includes a mapping from an arbitrary scenario language (e.g., an open scenario format such as OSC) or a proprietary scenario format such as a test specification for a HIL rig to an internal parameter format. That is, various scenario definition languages including SIL / HIL provided by OSC 2.0 as shown in block 105a, SIL / HIL as shown in block 105b, OSC 1.x, or a track (test case) as shown in block 105c can be mapped to a parameter space representation that includes one or more constraints, one or more values, or one or more relationships as shown in block 125.

[0036] Parallelized evaluation and / or asynchronous evaluation can provide maximum efficiency of information gain. Parallelized simulation can enable large-scale collection of information and subsequent generation of samples. Asynchronous evaluation can enable collection from hardware-in-the-loop (HIL) / on-vehicle test results to continue to notify the test. In some aspects, this technique can include a method for making decisions regarding further optimization using asynchronous information.

[0037] Accordingly, the simulation can be executed in a parallelized manner. The system can utilize parallelized assumptions to enable asynchronous updates of previous ones for sampling algorithms and parameter spaces. Specifically, the system can have the ability to enqueue new scenarios, build previous ones updated based on all the data collected so far, and use that previous one to enqueue the next scenario. Additionally, the integrated parameter space representation also enables the system to asynchronously update this parameter space performance using real-world data.

[0038] Using a parameter space representation as shown in block 125, a specific scenario as shown in block 135 can be generated. The specific scenario can be input into a parallelized test operation that includes various test modalities (e.g., SIL, HIL, track) as shown in block 145a. Results from the parallelized tests (e.g., SIL, HIL, track) can be collected for different test modalities as shown in block 145b. The results shown in block 145b can be fed back into the parameter space representation as shown in block 125 to continue processes such as generating further specific scenarios, further parallelized tests, and so on.

[0039] Various metrics can be used to evaluate coverage and performance across the scenario space. These scenarios can also enable safety queries and audits in a quantifiable way. And these aspects can enable a sampling algorithm to optimize parallelization in a way that maximizes statistical information metrics. Further, performance and coverage can be described using a single high-level statistically meaningful metric. The metric can include (a) an estimate of performance using confidence levels and confidence intervals with which the integrated parameter space can be normalized, and (b) an estimate of coverage based on sample density with which the integrated parameter space can be normalized, and (c) an integrated metric for the integrated parameter space, as well as a set of objective functions (e.g., boolean or continuous) and real-world observations.

[0040] Accordingly, the results shown in block 145b can be input into an evaluation metric operation as shown in block 155. The evaluation metric operation can be configured to use the parameter space representation to transform the test results in order to generate a performance / coverage metric as described.

[0041] Information from SIL, HIL, and track tests can be correlated to make inferences about performance in different situations. A conversion function between the performance, coverage, or metric distribution of SIL, HIL, and track tests and the corresponding uncertainty can be determined. A set of meta-statistics can quantify the amount of information loss during conversion between different test modalities and risky / uncertain regions based on this conversion. The set of meta-statistics can quantify the amount of tests preserved in each modality based on this correlation.

[0042] An exemplary process flow 100b for vehicle simulation, testing, and verification is provided in FIG. 1B. The scenario cross-compiler 110 can be configured to map from a scenario language (e.g., an open scenario format such as OSC, or a proprietary scenario format such as a test specification for an HIL rig) to an internal parameter format (e.g., an intermediate YAML format). The internal parameter format can be provided to a sampling service 120 configured to store and / or process one or more axes, one or more relationships, or one or more constraints calculated using the internal parameter format. The linker service 130 can create a test configuration object that can link between different objects, test results, etc. using the internal parameter format.

[0043] The linker service 130 can be configured to provide the test configuration object to a block 140 that includes one or more of a queue service, a parallel job scheduler, or an intelligent job manager. The queue service can submit the test configuration object to a plugin or service based on the satisfaction of the specific constraints. The parallel job scheduler can schedule jobs for different test modalities, simulators, etc. The intelligent job manager can determine different jobs to test based on sampling of a probability distribution.

[0044] Test results resulting from the combined operation of the queue service, parallel job scheduling, and intelligent job manager can be provided to block 150, which includes post-processing services, sampling services, data indexing, and feature quantification. The post-processing service can spin down the software. The sampling service, data indexing, and feature quantification can group the primary index with filtered values based on the feature vector. Data from block 150 can be provided to block 160, which includes an inference service and distribution post-processing. The inference service can filter relevant test-independent data into a multi-dimensional distribution. The distribution post-processing can generate an annotated data point probability distribution using the data point probability distribution.

[0045] The annotated data point probability distribution can be provided to block 170, which can include various other probability distributions (data point probability distributions specific to extended test modalities, annotated data probability distributions specific to test modalities). The probability distribution from block 170 can be provided to a learning correlation block 180 configured to generate a conversion function 190. The conversion function 190 can be configured to convert between different test modalities or other distributions. The data point probability distribution specific to the extended test modality can be provided to block 140 to continue the test process.

[0046] Internal representation A variety of industry-standard scenario formats or custom formats can be used for the simulation, testing, and validation of vehicles such as autonomous vehicles. A scenario-independent internal representation of the parameter space (e.g., an internal parameter format) and a mapping between this internal representation (e.g., an internal parameter format) and an open standard can enable a mapping between a variety of industry-standard scenario formats or custom formats and this internal representation (e.g., an internal parameter format). The internal representation of the parameter space (e.g., an internal parameter format) can enable the inclusion of information across one or more test modalities to facilitate additional information regarding the coverage and performance of the parameter space.

[0047] The internal representation of the parameter space (e.g., an internal parameter format) can include one or more components including: (i) one or more axes that can include one or more variables and possible values for the one or more variables, (ii) one or more relationships that can include one or more relationships between the one or more variables for the one or more axes, and one or more combinations between the one or more variables for the one or more axes, and (iii) one or more constraints that define one or more boundaries of a variable among the one or more variables and can define other regions within the scenario space that can be invalid.

[0048] Using this internal representation of the parameter space (e.g., internal parameter format), a wide variety of statistical operations can be performed in a scenario - and test - modality - independent manner. In particular, various statistical operations such as multidimensional clustering, nearest - neighbor analysis, parametric (or non - parametric) and / or linear (or non - linear) regression can be performed to inform the sampling algorithm more effectively compared to baseline sampling algorithms that do not use these various statistical operations. In particular, using various statistical operations, optimization algorithms and search algorithms can mutate the abstracted parameter space in a reversible manner so that they can operate on the search space more efficiently (e.g., with respect to computational complexity) compared to the parameter space where these various statistical operations are not performed. In particular, the application of these statistical operations can facilitate one or more of the following (when compared to the parameter space where these various statistical operations are not performed): (a) dimensionality reduction, (b) conversion from a non - convex search space to a convex search space, or (c) increasing the connectivity and / or continuity of the scenario space.

[0049] An internal representation of a parameter space (e.g., an internal parameter format) can be used by a device operable for one or more of vehicle simulation, vehicle testing, or vehicle verification. The device can comprise a memory and a processor operably coupled to the memory. The processor can be configured to execute instructions for causing the device to identify a first parameter set defined using a first scenario format. The first parameter set can include one or more of the following: (i) one or more first scenario format axes, (ii) one or more first scenario format relationships, or (iii) one or more first scenario format constraints. The processor can be configured to execute instructions for causing the device to map the first parameter set to an internal parameter set using an internal parameter format, the internal parameter format can include one or more of the following: (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints. The processor can be configured to execute instructions for causing the device to generate a test configuration object using the internal parameter set. The test configuration object can be configured to link to one or more objects. The processor can be configured to execute instructions for causing the device to send the test configuration object to a test modality for testing.

[0050] A process flow 200 for vehicle simulation, testing, or verification can be provided as shown in FIG. 2. Inputs (e.g., test specification 210) related to scenarios of a first scenario format (e.g., scenario definition language) can be identified. The test specification 210 can include one or more of a hardware-in-the-loop (HIL) test specification, a software-in-the-loop (SIL) test specification, or a field test specification. The HIL test specification can include, for example, a dSpace® HIL specification. The SIL test specification can be implemented using a first scenario format (e.g., scenario definition language) such as OSC 1.0, OSC 2.0, different scenario languages, etc.

[0051] In one example, if the scenario is outside a particular scenario language, the scenario can be imported or converted into the particular scenario language. If it is in the particular scenario language, the test specification 210 can be cross-compiled into an internal parameter format that can be in yet another intermediate markup language (YAML) format, as shown in operation 220 which can be performed in a one-to-one operation. The internal parameter format can include one or more of the following: (i) one or more internal axes, (ii) one or more internal relationships, (iii) one or more internal constraints, (iv) one or more internal metrics, or (v) one or more parse operation information that facilitates the inverse conversion to a first scenario format (e.g., a particular scenario language).

[0052] In another example, if the scenario is in a particular scenario language, the internal parameter format can be configured to convert from the particular scenario language to the internal parameter format without importing or converting the scenario into the particular scenario language. The internal parameter format can include additional conversion data that can be used to facilitate the conversion from the particular scenario language to the internal parameter format.

[0053] As a result, cross-compiling the test specification 210 into the first parameter set of the internal parameter format can cause one or more internal axes, one or more internal relationships, or one or more internal constraints to be parsed into the internal parameter format, as shown in operation 230 which can be performed in a one-to-one operation. The internal parameter format (e.g., YAML format test specification) can be exchangeable between various test modalities (e.g., HIL, SIL, field test, etc.) and simulators (e.g., HIL simulator, SIL simulator, field test simulator, etc.). This internal parameter format can include one or more parse operation information (e.g., related links and context information) used to identify the compilation source of the internal parameter format in order to facilitate reverse tracing the internal parameter set of the internal parameter format to the first parameter set of the first parameter format (e.g., a specific scenario language for identifying information specific to a particular scenario language). The internal parameter format can include one or more internal metrics for generating metrics using axes when the results are available after testing.

[0054] Using the first parameter set of the internal parameter format, one or more of a test configuration object, a test run specification, or a test run result can be generated. Using the first parameter set of the internal parameter format, e.g., an intermediate YAML format, a test configuration object (i.e., the test configuration object) can be generated as shown in operation 240 which can be a one-to-one operation. The test configuration object can include references to one or more of a test specification, a parameter set of the internal parameter format, a test result, etc. to facilitate indexing of data related to a specific test.

[0055] To start a test, send a request (and associated links) that includes a first parameter in an internal parameter format, as shown in operation 250 which can be a one-to-many operation, to a microservice to enqueue a test job and generate a test run specification. The test run specification can be a data package that contains information used to execute a test in a test environment. This information that can be used to execute a test in a test environment can include one or more of the following: (i) a test environment specification, (ii) references to one or more input files such as a test specification file, (iii) runtime parameters, etc. A test configuration object can be formatted into a test modality format (e.g., a format specific to SIL, HIL, track, etc.) for a test modality (e.g., SIL, HIL, track, etc.). The formatted test configuration object can be sent to a test modality (e.g., SIL test, HIL test, track test, etc.) associated with the test modality (e.g., SIL, HIL, track, etc.).

[0056] Requirements for generating test run specifications based on the required test modalities can be transferred to one or more interfaces for one or more test modalities, including: (i) an interface for a specific test scenario simulator, (ii) a plug-in interface for one or more of an external SIL simulator or an external HIL simulator, or (iii) a plug-in interface for an automated track test job queue. These one or more interfaces can perform further formatting from an internal parameter format (e.g., an intermediate scenario language) to a test specification. The further formatting can include one or more of the following: HIL rig specifications, track test specifications, SIL specifications, etc. These interfaces (e.g., plug-in interfaces) can submit the requirements to the corresponding test modality (e.g., a SIL test modality for SIL test specifications, a HIL test modality for HIL test specifications, a track test modality for track test specifications, etc.) via an available web application protocol interface (API) (e.g., Representational State Transfer (REST), Hypertext Transfer Protocol (HTTP), etc.).

[0057] As shown in operation 260, the test run specification operation 250 can generate a test run result that can be a one-to-one operation. The test run result can include one or more of drive data, simulation data, or additional data that can be parsed based on the axis type.

[0058] Test modality format data can be converted into internal parameter format data in an internal parameter format converter plugin. In one example, external test run results (e.g., test run results for a particular scenario language) can be received using the converter plugin. The external test run results can include various test modalities including one or more of SIL, HIL, drive, etc. The converter plugin can receive various inputs including: (i) internal parameter format, (ii) drive logs in any format, etc. The converter plugin can define a conversion from drive log format to internal parameter format. Metrics in the internal parameter format can be identified after testing.

[0059] Using the test run results, test-independent data points can be generated using a one-to-many operation as shown in operation 270. The test run results, which can include drive data, simulation data, etc., can exist in a time-based metric format. The time-based metric format can be collected from various test modalities such as SIL, HIL, drive, track, etc. The time-based metric format can be converted into an internal parameter format (e.g., intermediate YAML format). The time-based metric format can be converted into an internal parameter format based on one or more of the following: (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints.

[0060] The internal parameter format converter plugin can be configured to convert axis data in a time-based metric signal format into internal parameter format data and write the axis data to a database. In one example, a time-based metric format can be converted into an intermediate YAML format using a YAML template for the axis to generate an intermediate YAML format metric. A post-processing microservice can calculate relevant axis values from these intermediate YAML format metrics that can be written to the database. Any axis that violates a constraint or relationship specified in the intermediate YAML format can be updated in the intermediate YAML format to include input axis values that can be identified as untested.

[0061] Using test-independent data points that can be generated as shown in operation 270, a data point probability distribution can be generated as shown in operation 280, which can be a many-to-many operation. The data point probability distribution can be generated using one or more of the following: (i) an axis referenced in a particular data point probability distribution, (ii) a value referenced in a particular data point probability distribution, or (iii) a parameter referenced in a particular data point probability distribution.

[0062] The scenario definition language may include various scenario languages (e.g., open scenario formats such as OSC, or proprietary scenario formats such as test specifications for HIL rigs). In one example, the scenario definition language may be an open scenario language. When the open scenario language is used, a simulator (e.g., an autonomous vehicle (AV) simulator) may be configured to receive scenario format data and convert the scenario format data into internal parameter format data (e.g., intermediate YAML format data as shown in operation 220). The vehicle simulator may be configured to generate test configuration objects using the internal parameter format data (e.g., as shown in operation 240) and test the test configuration objects using a test modality (e.g., as shown in operation 250). The vehicle simulator may be configured as otherwise disclosed with respect to generating an internal representation of the parameter space provided herein.

[0063] The vehicle simulator may be configured to generate various metrics (e.g., as shown in operation 260) for measuring one or more of the coverage or performance of the vehicle simulator. The vehicle simulator may be configured to calculate parametric metrics (e.g., metrics calculated in parametric format). The parametric metrics may be based on one or more of parametric performance metrics or parametric coverage metrics. The parametric metrics may be selected based on a particular use case type. The vehicle simulator may be configured to extract one or more axis values for a test modality and calculate one or more axis-specific metrics based on the one or more axis values. The vehicle simulator may be configured to generate test configuration objects based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metrics.

[0064] Parallel simulation using asynchronous information The functionality for vehicle simulation, testing, or verification can be configured to generate parallel simulations using asynchronous information. Combining asynchronous information with parallelized simulations can enhance the information gain over a specific time for a selected amount of computing resources. The parallelized simulations can facilitate large-scale collection of information and subsequent generation of samples. Updates to asynchronous information can facilitate collection from various test modalities (e.g., SIL, HIL, track) for determining subsequent tests. Tests that combine parallel job scheduling with asynchronous information can enhance the information gain when compared to a baseline approach, as it is possible to update the currently running parallel tests using partial information without random scheduling of subsequent test jobs and without waiting for complete information before adjusting subsequent test jobs.

[0065] In addition, adding, adjusting, or terminating different test samples based on asynchronous information before completion of the tests for a specific test batch can reduce the computational complexity for achieving convergence for a specific test case when compared to the baseline computational complexity where asynchronous information is not used, parallel tests are not used, or neither asynchronous information nor parallel tests are used. The reduction in computational complexity for achieving convergence for a specific test case can occur due to an increase in the tests for one or more of the high-sensitivity cases, undetermined cases, edge cases, or cases prone to failures, and a decrease in the tests for cases with lower information that provide lower information when compared to the baseline test cases. For example, cases with lower information can include low-sensitivity cases, high-coverage cases, redundant test cases, cases with appropriate test coverage, low-failure cases, etc.

[0066] The sampling algorithm and operations on the parameter space can be updated asynchronously using parallelized assumptions. That is, when a scenario is queued, based on the data collected, an updated previous one can be generated to queue subsequent scenarios. A scenario can include data collected asynchronously using an internal parameter format. The data collected asynchronously can include one or more of simulated data or real-world data.

[0067] The vehicle simulator can be configured for one or more of simulation, testing, or verification as provided in process flow 300a in FIG. 3A. The vehicle simulator can be configured to test a first test sample batch in a test modality format. The first sample batch can include one or more first test samples for outputting one or more first test sample results. The vehicle simulator can be configured to identify one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed. The vehicle simulator can be configured to convert one or more asynchronous test sample results from a test modality format to an internal parameter format. The vehicle simulator can be configured to adjust a second test sample batch based on one or more asynchronous test sample results.

[0068] The vehicle simulator may be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved. For example, after testing the first and second batches, the coverage metric may not have achieved a selected threshold. In this example, the vehicle simulator may be configured to adjust subsequent test sample batches (e.g., the third, fourth, etc.) until the coverage metric achieves the selected threshold. Testing of subsequent test sample batches may be repeated until the selected coverage metric is achieved.

[0069] The vehicle simulator may be configured to process test specifications for transmission to the queue service 340a using one or more operations. The test specifications may include one or more of, as shown in block 302, HIL test specifications, OSC™ 1.0 or OSC™ 2.0 test specifications, field test specifications, etc. The test specifications may be provided to a scenario cross-compiler 310 that may compile the test specifications into an internal parameter format (e.g., an intermediate YAML representation), as shown in block 312.

[0070] The internal parameter format (e.g., an intermediate YAML representation) may be provided to one or more of the sampling service 320 or the linker service 330. The sampling service 320 may generate an object representation of one or more of one or more axes (e.g., a numerical axis or a discrete axis), one or more relationships, or one or more constraints, as shown in block 322. The linker service 330 may receive the internal parameter format (e.g., a YAML representation) and generate a test configuration object, as shown in block 332, that may provide links between one or more objects, results, etc. A test configuration object as shown in block 332 may be provided to the queue service 340a.

[0071] As shown in block 332, the queue service 340a can receive a test configuration object, and as shown in block 338, can receive user input (e.g., user requests). The queue service 340a can submit the test configuration object as shown in block 332 and the user input as shown in block 338 to one or more of the plugins or services for execution based on the satisfaction of one or more constraints. A test configuration object such as that shown in block 342 can be provided to one or more of the HIL plugin 344a, SIL plugin 344b, field test plugin 344c, or test run specification 346.

[0072] The queue service 340a can receive a test trigger 348 from the intelligent job manager 340c. The intelligent job manager 340c can be configured to sample a data probability distribution (e.g., an extended data probability distribution) for one or more test cases for a particular test modality (e.g., SIL, HIL, track, etc.). One or more test cases can be one or more of a sensitivity case, an indeterminate case, an edge case, or a case where a failure is likely to occur. A sensitivity case can be a test case based on a variable that can have a large impact on the result when compared to a baseline test case. A baseline test case can be a test case that can be based on one or more of an average test sample result, a median test sample result, etc. An indeterminate case can be a test case that does not include sufficient test sample results to provide sufficient information about coverage, performance, or other suitable metrics. An edge case can be a test case that tests the boundary between one or more different test cases. A case where a failure is likely to occur can be a test case that is more likely to result in a failure when compared to a baseline test case.

[0073] The Intelligent Job Manager 340c can be configured to identify previous test sample results, currently running test samples (e.g., test jobs, simulations, etc.), and the overall probability space of different test cases. The Intelligent Job Manager 340c can receive test sample results that can change the data probability distribution. The test sample results can include asynchronous test sample results. The Intelligent Job Manager 340c can be configured to receive and / or identify asynchronous test sample results from a first batch of test samples in a specified test modality format before the first batch of test samples completes the test. The Intelligent Job Manager 340c can be configured to convert asynchronous test sample results to an internal parameter format when the asynchronous test sample results are in a test modality format (e.g., SIL, HIL, track, etc.).

[0074] The Intelligent Job Manager 340c can be configured to adjust a second batch of test samples based on one or more asynchronous test sample results. For example, the Intelligent Job Manager 340c can cancel the currently running test sample in the first batch of test samples to free up computing resources for other tests when the unique test sample results associated with the currently running test sample are not used to change the data probability distribution based on asynchronous test sample results that can change the data probability distribution. The cancellation request can be sent from the Intelligent Job Manager 340c to the parallel job scheduler 340b (as shown in block diagram 300b in Figure 3B).

[0075] The intelligent job manager 340c can be configured to down-weight the currently executing test samples when sampling the data probability distribution in order to maximize information gain and minimize redundant tests. The currently executing test samples may not have test sample results included in the data probability distribution, but the currently executing test samples may have been previously enqueued. Thus, to avoid double counting of the currently executing test samples, the data probability distribution can be adjusted (e.g., down-weighted) based on the currently executing test samples.

[0076] The vehicle simulator can be configured to process test specifications for generating test run results 358 using one or more operations, as shown in FIG. 3B. The test run specification 346 (such as provided by one or more of the queue service 340a, HIL plugin 344a, SIL plugin 344b, or field test plugin 344c) can be provided to the parallel job scheduler 340b.

[0077] The parallel job scheduler 340b can be configured as follows: (i) test one or more test sample batches in various test modality formats, and (ii) send the test sample batches to various test modalities. The test sample batch can include one or more test samples that can be calculated to output one or more test sample results.

[0078] The parallel job scheduler 340b can be configured to identify various parameters related to job scheduling, including one or more of randomness, decreasing learning rate, predicted sample size, test batch configuration parameters (e.g., hyperparameters related to past, current, and predicted future test executions). The parallel job scheduler can be implemented using various algorithms that can include hyperparameters based on the operating parameters of the algorithm.

[0079] The parallel job scheduler 340b can be configured to select a test sample batch and provide that test sample batch in the test run specification 352 to one or more of the following: (i) a test modality format (e.g., HIL simulation 356a, SIL simulation 356b, field test 356c) for formatting into a test modality format (e.g., HIL, SIL, field test, etc.) that is input into a test modality format, or (ii) one or more test modality plugins (e.g., HIL plugin 354a, SIL plugin 354b, field test plugin 354c) as input for a specific simulation 356d test. Results from one or more of the HIL simulation 356a, SIL simulation 356b, field test 356c, applied simulation 356d test, etc. can be collected as test run results 358. These test run results can be provided to the job post - processing service 450a (as shown in FIG. 4A).

[0080] The parallel job scheduler 340b can be configured to schedule one or more of jobs, simulations, test batches, etc. to different test modalities, simulators, etc. by allocating software, network, or other computing infrastructure for operating jobs, simulations, test batches, etc. in parallel. The intelligent job manager 340c can be configured to send requests 359 to the parallel job scheduler 340b to intersect and cancel jobs to free up computing resources. The intelligent job manager 340c can be configured to generate test specifications (e.g., newly generated test specification 311) that are processed as test specifications by being input into a scenario cross - compiler 310 (as shown in FIG. 3A).

[0081] The parallel job scheduler 340b can be configured to receive requests 359 for intersecting and canceling jobs to free up computing resources. The requests 359 for intersecting and canceling jobs to free up computing resources can be based on asynchronous test sample results. As a result, the parallel job scheduler 340b can adjust subsequent test sample batches based on the asynchronous test sample results. Alternatively, or in addition, the parallel job scheduler 340b can be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0082] The test run results 358 can include various metrics, axis values, etc. The test run results 358 can include one or more of a stack log or an environment log. The test run results 358 can include various data files such as sensor and rendering data files. The test run results 358 can be provided to the post-processing service 350a. The post-processing service 350a can be configured to convert axis data in a time-based metric signal format to internal parameter format data in an internal parameter format converter plug-in. The axis data, which can be in a time-based metric signal format or an internal parameter format, can be written to a database. The database can be used when retrieving one or more axis values for a test modality (e.g., SIL, HIL, track) and when calculating one or more axis-specific metrics based on the one or more axis values. The data storage in the internal parameter format can facilitate efficient data retrieval when performing calculations. Alternatively, or in addition, the calculations can be vectorized to enhance the calculation efficiency.

[0083] The test run result 358 can be processed by various operations to generate a data probability distribution to be sent to the intelligent job manager 340c. The data probability distribution can be calculated using parametric metrics (e.g., metrics based on one or more parameters). The parametric metrics can be calculated using one or more of parametric performance metrics or parametric coverage metrics. The parametric metrics can be based on a specific use case type.

[0084] Parametric metrics (e.g., tunable exploration metrics) can be used to explore the information space to be tested. In one example, the parametric metrics can be tuned based on the difference between the random samples and the collected samples. For example, for a uniform discrete distribution from 1 to 10, a heavy weighting of samples near the upper end of the range can indicate that samples near the lower end of the range can be explored. The tunable exploration metrics can be adjusted when samples near the lower end of the distribution are collected. That is, using the tunable exploration metrics, the degree of samples outside the distribution can be determined and adjusted when the distribution changes in response to sample collection.

[0085] Alternatively, or in addition, the parallel job scheduler 340b may be configured to generate subsequent test sample batches based on one or more axis-specific metrics. The one or more axis-specific metrics can be based on one or more axis values selected based on parametric metrics.

[0086] The parallel job scheduler 340b may be configured to schedule jobs using asynchronous information (e.g., asynchronous test results). The parallel job scheduler 340b may be configured to schedule one or more different test samples in parallel and use asynchronous information (e.g., asynchronous test results) to end, adjust, or add different test samples. The parallel job scheduler 340b may use asynchronous information (e.g., asynchronous test results) to end, adjust, or add different test samples without having a completion time for a particular test sample. The parallel job scheduler 340b may receive additional test samples for testing based on asynchronous information (e.g., asynchronous test results) before a previous test sample finishes testing.

[0087] A device for testing, simulating, or validating a vehicle may include a memory and a processor. The processor may be configured to execute instructions to cause the device to test a first test sample batch including one or more first test samples and output one or more first test sample results. The plurality of first test sample results may include completed test samples (which may be used as asynchronous information) and uncompleted samples (and thus may not be used as asynchronous information). The plurality of first test sample results may be generated using one or more different test modalities (e.g., SIL, HIL, track, etc.).

[0088] The processor may be configured to identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed. That is, the asynchronous test sample results may be the results of completed test samples even if the first test sample batch further includes incomplete test sample results. The asynchronous test sample results may be based on real-world data. One or more asynchronous test sample results may be used to calculate parametric metrics.

[0089] The parametric metric may include one or more of a parametric performance metric or a parametric coverage metric. The first sample batch may be terminated before completion based on the parametric metric. The parametric metric may be used to adjust a second test sample batch. The second test sample batch may be adjusted based on test adjustment configuration parameters including one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, etc.

[0090] The parallel job scheduler 340b may be configured to convert a plurality of first test samples from an internal parameter format to a test modality format. For example, the internal parameter format may be converted to a HIL format (e.g., using a HIL plugin), to a SIL format (e.g., using a SIL plugin), or to a field test format (e.g., using a field test plugin).

[0091] A plurality of first test samples may be in a test modality format when asynchronous information is identified. Asynchronous information (e.g., one or more asynchronous test sample results) may be converted from the test modality format to the internal parameter format (if present in the test modality format). Converting the asynchronous information to the internal parameter format may minimize the computational complexity, computational time, and cost of adjusting the second test sample batch based on the asynchronous information.

[0092] Metric A device for vehicle simulation, vehicle testing, or vehicle verification may include a memory and a processor operably coupled to the memory. The processor may be configured to execute instructions to cause the device to identify a first set of parameters defined using a first scenario format. The first set of parameters may include one or more of one or more first scenario format axes, one or more first scenario format relationships, or one or more first scenario format constraints.

[0093] As shown in FIG. 4A, functionality 400a is provided that may be configured such that a post - processing service 450a identifies a first set of parameters. The post - processing service 450a may be configured to map the first set of parameters to an internal set of parameters. The internal set of parameters may include one or more of one or more internal axes, one or more internal relationships, or one or more internal constraints. The post - processing service may be configured to transform the first set of parameters (e.g., test run results) to generate an internal set of parameters (e.g., test - independent data point 451). The internal set of parameters (e.g., test - independent data point 451) may be input to a sampling service 450b.

[0094] The sampling service 450b may be configured to calculate asynchronous information provided to an intelligent job manager 340c (such as that shown in FIG. 3A, for example). The sampling service 450b may be configured to identify one or more asynchronous test sample results of one or more first test samples before a first test sample batch finishes testing. The sampling service 450b may be configured to calculate parametric metrics. The parametric metrics may be based on one or more of a parametric performance metric or a parametric coverage metric. The sampling service 450b may be configured to select a set of algorithms based on the parametric metrics. The sampling service 450b may be configured to apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch finishes testing.

[0095] The sampling service 450b may be configured to generate and adjust an information space including an internal parameter set by adding a data index based on one or more of: (i) one or more internal axis values, (ii) one or more internal relationship values, or (iii) one or more internal constraint values.

[0096] The sampling service 450b may be configured to select one or more of a set of axes based on parametric metrics, or a subset of the set of axes. Results based on one or more of the set of axes based on parametric metrics, or a subset of the set of axes, may be displayed on a display device. One or more axis values for the set of axes or the subset of the set of axes may be retrieved for a particular test modality (such as HIL, SIL, track, etc.), and one or more axis-specific metrics may be calculated based on the set of axes or the subset of the set of axes.

[0097] The sampling service 450b may be configured to calculate parametric metrics. The parametric metrics may be based on one or more of parametric performance metrics or parametric coverage metrics. The parametric metrics may be based on a specific use case type. The sampling service 450b may be configured to generate test configuration objects based on one or more axis-specific metrics. The one or more axis-specific metrics may be based on one or more axis values selected based on the parametric metrics. Results based on the parametric metrics may be displayed on a display device.

[0098] As shown in FIG. 4B, various operations 400b indicate that the sampling service 450b may send test-independent data points to a data indexing block 460a configured to process the test-independent data points by constructing a primary index (as shown in block 462) and constructing a secondary index (i.e., a view) (as shown in block 464). The primary index may include one or more of the following: (a) date, (b) axis, (c) test environment, (d) stack version, (e) other metadata, etc. (as shown in block 463).

[0099] The secondary index of block 464 that constructs the secondary index may be activated based on user input or based on the availability of computing resources. Block 465 that constructs the secondary index may generate a group of primary indexes that may include filtered values configured to select specific values, ranges, etc. of the test-independent data points.

[0100] The test-independent data point 461 can be sent to a feature quantification block 460b that can process the test-independent data point 461 to generate results that are sent to one or more of the following in a feedback loop: (i) the test-independent data point 461, block 463, or block 465. As shown in FIG. 4C, a further operation 400c can include sending the test-independent data point 461 to a feature quantification block 460b that can be input to a block 466 that applies a kernel function. A kernel function can be applied to the test-independent data point to generate a feature vector 468. Using various techniques, a feature vector 468 can be generated that includes one or more of the following: (i) linear, (ii) non-linear, (iii) deep learning-based functions, (iv) frequency space transforms, (v) other transforms, etc., as shown in block 467. The output of the feature quantification block 460b (e.g., the feature vector 468) can be sent to a data indexing block 460a (e.g., the test-independent data point 461, block 463, or block 465).

[0101] As shown in FIG. 4A, the sampling service 450b can use data indexing in block 460a and feature quantification 460b to generate a test-independent data point and index 453. The test-independent data point and index 453 can be sent to an inference service 450c. The inference service 450c can be configured to filter the test-independent data point into a multi-dimensional distribution. The test-independent data point and index 453 can be filtered into a multi-dimensional distribution based on one or more of relevance or various data point spaces that can generate a data point probability distribution 480 using one or more primary indexes, secondary indexes, features, etc. The data point probability distribution 480 can be sent for distribution post-processing 460c.

[0102] A user interface may be provided to enable a user to view at a glance one or more of vehicle stack performance, test coverage, or uncertainty / test gap via a web-based viewer. Via the user interface, the user may select any axis or combination of axes (e.g., performance metrics such as pass rate, collision margin time, variables such as weather, driving speed, road type, etc.). A multi-dimensional chart of test results related to a set of axes may be displayed via the user interface (e.g., scatter plot matrix, cluster scatter plot, surface / contour plot, etc.) based on the selection on the user interface. Via the user interface, filters and switch axes may be applied to these charts to iteratively filter the information space (e.g., stack performance, test coverage, uncertainty / test gap, etc.).

[0103] In some examples, the user may click on recommendations for coverage gaps, highly correlated axes, relationships, clusters, and high uncertainty, and low metric values. The user may send a request to the sampling service 450b to select one or more combinations of axes, grouping, clustering, smoothing, interpolation, or statistical metrics for viewing. The sampling service 450b may attempt to retrieve axis values for a particular test modality (e.g., SIL, HIL, track, etc.). When there are no axes for a particular test modality, the axis values may be calculated as requested, and the sampling service 450b may provide one or more of vehicle stack performance, test coverage, test gap, etc. to the user interface. Alternatively, or in addition, the sampling service 450b may provide one or more of vehicle stack performance, test coverage, test gap, etc. to the user interface when there are axes for a particular test modality.

[0104] When the axis values are calculated, the distribution post - processing 460c operation can be configured to facilitate various operations for sending data to the probability distribution 470 operation and receiving the data point probability distribution 480, as shown in operation 400d illustrated in FIG. 4D. As shown in block 469a, the data point probability distribution 480 can be extended with a kernel function, or as shown in block 469b, a derived distribution can be calculated using the data point probability distribution 480.

[0105] When the data point probability distribution 480 is extended with a kernel function, a data point probability distribution 469c can be calculated. When a derived distribution is calculated using the data point probability distribution 480, one or more of the gradient 469d or the density 469e can be calculated. As shown in block 469f, one or more of the data point probability distribution 469c, the gradient 469d, or the density 469e can be used to perform a statistical analysis.

[0106] The performed statistical analysis can include any suitable technique for generating a probability distribution including one or more of the following: (i) clustering, (ii) regression, (iii) importance (e.g., based on a combination of coverage and / or performance), (iv) sensitivity analysis, or (v) entropy, as shown in block 469g. The distribution post - processing 460c operation can provide an input to the probability distribution 470 operation for further processing to the intelligent job manager 440c.

[0107] The statistical analysis performed may include any suitable analysis technique for analyzing the data point probability distribution. The statistical analysis performed may include grouping combinations of axis values for further calculations based on one or more of metadata tags, scenarios, drive tags, locations, etc. Alternatively, or in addition, derived metrics (e.g., reliability, confidence interval, mean, minimum, maximum, count, standard deviation, statistical entropy, etc.) may be calculated. Alternatively, or in addition, clustering and correlation analysis may be performed to determine patterns. Exemplary patterns may include linear kernel separators, kernel-based correlation analysis, K-means, DB-SCAN clustering, etc. Alternatively, or in addition, outliers, clusters, and associated metrics may be linked to these results. Alternatively, or in addition, smoothing and interpolation may be performed to include one or more of Gaussian smoothing, linear interpolation, nearest neighbor interpolation, cubic interpolation, quintic interpolation, etc.

[0108] The user interface may provide page filtering for specific scenarios, simulations, drives, etc. Inputs to the user interface may be submitted to a scenario cross-compiler 310, as shown in FIG. 3A, and compiled into scenarios and test cases for a specific test modality (e.g., HIL, SIL, track, etc.). A plug-in specific to the test modality may be configured to compile an internal parameter set (in an internal parameter format) into a test modality format that may include a basic external scenario.

[0109] As shown in block diagram 500 in FIG. 5, the output from the distributed post - processing block 560c can be sent to a probability distribution block 570 (e.g., the annotated data probability distribution block 571). The annotated data probability distribution block 571 can be, for example, a data probability distribution that can be used to calculate metrics using an internal parameter set. This metric can include one or more of a performance metric or a coverage metric. This metric can be based on the use - case type. This metric can be calculated using one or more of a confidence level, a confidence interval, a normalized format, or a sample density.

[0110] The annotated data probability distribution block 571 can be configured to identify a particular driving situation based on one or more of the following: (i) importance (e.g., high sensitivity), (ii) density with respect to importance (e.g., additional data collection can be used), (iii) correlation variables (e.g., regression), (iv) clustering (e.g., discrete patterns), etc. These metrics (e.g., importance, density with respect to importance, correlation variables, clustering, etc.) can be used to select axes for additional testing.

[0111] The metric can be an integrated metric that can be calculated based on performance metrics and coverage metrics. The integrated metric can be based on one or more of an objective function (e.g., boolean or continuous) or real - world data. The integrated metric can be a single metric that facilitates the determination of performance and coverage for an integrated parameter space. That is, the integrated metric can be used for one or more of the verification or auditing of one or more of performance or coverage with respect to the integrated parameter space. In one example, a query includes: "Does my stack prevent collisions in hard - brake events with a 99.99% success rate and a 0.01% error range?" In response to the query, a set of statistical metrics (e.g., metric, metric, integrated metric, etc.) that accept or reject the query can be returned.

[0112] Vehicle safety characteristics can be queried and / or audited using quantification. Using the annotated data probability distribution block 571, vehicle safety characteristics can be queried, and in response to the vehicle safety characteristics query, a safety metric can be calculated based on a metric.

[0113] Parallelization can be used to calculate a metric to maximize one or more of the operational design domain (ODD) coverage or ODD performance. Using the annotated data probability distribution block 571, the ODD coverage for an autonomous vehicle can be determined based on a metric. Using the annotated data probability distribution block 571, the ODD performance for an autonomous vehicle can be determined based on a metric. That is, a metric can be used to evaluate the coverage and / or performance of a set of tests across the target ODD of a vehicle stack (e.g., an autonomous vehicle stack). A metric can be used to select one or more sampling algorithms to optimize parallelization to maximize subsequent metrics (e.g., metrics, integrated metrics, etc.).

[0114] A metric can be used to determine specific tests that can be used to facilitate the collection of data to maximize one or more of the performance or coverage of an integrated parameter space. In one example, the metric can indicate that light detection and ranging (LIDAR) can be used to increase one or more of the performance or coverage of an integrated parameter space. In another example, a user interface can receive an input that can select a specific test based on a specific use case. To facilitate data collection, a selected number of different algorithms and / or metrics associated with one or more tests can be provided. The user interface can receive an input to select one or more of a specific test (e.g., LIDAR), a specific algorithm, a specific metric, etc.

[0115] Metrics can be used with an internal parameter format to minimize data collection time and computing resources and to maximize the quality of the collected data. A device for vehicle testing may include memory and a processor operably coupled to the memory. The processor may be configured to execute instructions to cause the device to identify a first set of parameters defined using a first scenario format, the first set of parameters including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to execute instructions to cause the device to map the first set of parameters to an internal set of parameters including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to execute instructions to cause the device to calculate a metric based on one or more of a performance metric or a coverage metric. The device may include a display device configured to display a result based on the metric. The metric may be based on a use case type.

[0116] Using the annotated data probability distribution block 571, metrics can be provided to a sampling service (e.g., sampling service 450b as shown in FIG. 4A) and / or an inference service (e.g., inference service 450c as shown in FIG. 4A). The sampling service 450b and / or the inference service 450c can be configured to perform one or more of the following: (i) select a set of axes based on the metrics, (ii) display results based on the set of axes on a display device, (iii) select a subset of the set of axes, or (iv) display a subset of the results based on the subset of the set of axes on a display device. The sampling service 450b and / or the inference service 450c can be configured to extract one or more axis values for one or more test modalities and calculate one or more axis-specific metrics based on the one or more axis values. For example, axis-specific metrics based on one or more axis values based on the metrics can be sent to the scenario cross-compiler 310 for transmission to the linker service 330 for generating test configuration objects.

[0117] The metrics can be used with asynchronous test sample results to minimize test time and computational resources and maximize one or more of the coverage or performance of subsequent test results. A device for vehicle simulation, testing, or verification can include a memory and a processor operably coupled to the memory. The processor can be configured to execute instructions for causing the device to perform one or more of the following: (i) test a first test sample batch including one or more first test samples and output one or more first test sample results, (ii) identify one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed, (iii) calculate a metric including one or more of a performance metric or a coverage metric based on the one or more asynchronous test sample results, or (iv) adjust a second test sample batch based on the metric.

[0118] The processor may be configured to determine one or more asynchronous test sample results based on parametric metrics that may include one or more of parametric performance metrics or parametric coverage metrics. The asynchronous test sample results may be determined based on real-world data.

[0119] The first test sample result may be generated using one or more different test modalities (e.g., SIL, HIL, track, etc.). The first test sample may be converted from an internal parameter format to a test modality format. The asynchronous test sample result selected from the first sample batch may be converted from the test modality format to the internal parameter format.

[0120] Using the metric, the first sample batch may be terminated before the first sample batch completes the test. Using the metric, the second test sample batch may be adjusted based on test adjustment configuration parameters including one or more of the selected sampling algorithm, randomness parameter, or other hyperparameters such as learning rate decay parameter or sample size parameter.

[0121] As shown in FIG. 5, the annotated data probability distribution block 571 may be sent to the test modality-specific annotated data probability distribution block 572, and the annotated data probability distribution block 572 may be configured to generate various test modality-specific distributions and sub-distributions including: (i) the SIL-specific annotated distribution 573a, (ii) the HIL-specific annotated distribution 573b, (iii) the track-specific annotated distribution 573c, etc. These test modality-specific annotated distributions (e.g., 573a, 573b, 573c) may be sent to the extended test modality-specific data probability distribution block 574 for transmission to the intelligent job manager 540c.

[0122] The extended test modality-specific data probability distribution block 574 can include one or more of one or more distributions or sub-distributions such as SIL, HIL, drive data, etc. The extended test modality-specific data probability distribution block 574 can be extended by various tests performed using the conversion function 590 between two data probability distribution blocks by including factors related to the increased variance and / or uncertainty resulting from the conversion.

[0123] The annotated data probability distribution block 571 can be configured to send the annotated data probability distribution to the learning correlation block 580 for processing. The output from the learning correlation block 580 can be sent to the conversion function 590 between two data probability distribution blocks, which can be configured to output to the extended test modality-specific data probability distribution block 574.

[0124] Conversion Function between Test Modalities Devices for vehicle testing, simulation, and verification can include a memory and a processor. The processor can be configured to execute instructions for causing the device to identify first test data based on a first test modality, and the test data can include one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data. For example, the learning correlation block 580 can be configured to receive data from the annotated data probability distribution block 571 as shown in FIG. 5. As shown in FIG. 6, the data provided from the probability distribution block 670 to the learning correlation block 680 can be one or more extended data point probability distributions (e.g., extended data point probability distribution 1, 681a, and extended data point probability distribution 2, 681b).

[0125] A conversion function can be generated to switch between different test data modalities. The processor can be configured to execute instructions for causing the device to compute a conversion function configured to convert test data from a first test modality to a second test modality different from the first test modality. For example, the cross-distribution inference block 682 can be configured to generate the following conversion functions 686a, 686b: (i) a conversion function from the extended data point probability distribution 1, 681a to the extended data point probability distribution 2, 681b, and (ii) a conversion function from the extended data point probability distribution 2, 681b to the extended data point probability distribution 1, 681a. The conversion functions 686a, 686b can be generated using any suitable function including one or more of the following: (a) a non-parametric function 683 (e.g., a kernel), (b) a parametric function 684, or (c) a deep learning-based learning function 685.

[0126] Using the conversion function, test data of a test modality different from the test modality of the received test data (e.g., the first test data of the first test modality) can be computed (e.g., the second test data of the second test modality). The processor can be configured to use the conversion function to compute second test data including one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data. The first test modality can be any suitable test modality including one or more of the following: SIL, HIL, track test, etc. The second test modality can be any suitable test modality including one or more of the following: SIL, HIL, track test, etc.

[0127] Calculate some metrics and compare the differences between the first test data and the second test data that can result as a result of generating the second test data using the first test data and a conversion function between the first test data and the second test data. These metrics can include: (i) an information loss metric for determining the amount of information loss that occurs when the first test data is converted to the second test data using the conversion function, (ii) a risk metric for determining the amount of risk that occurs when the first test data is converted to the second test data using the conversion function, or (iii) an uncertainty metric for determining the amount of uncertainty that occurs when the first test data is converted to the second test data using the conversion function.

[0128] Using some of the metrics, it can be determined whether there is a reduction in one or more of the test time or computational complexity that can occur when the conversion function is used compared to when the conversion function is not used. The processor can be configured to calculate one or more of a conversion test time or a conversion computational complexity for calculating the second test data using the conversion function based on the first test data. The processor can be configured to calculate one or more of a non-conversion test time or a non-conversion computational complexity for calculating the second test data without using the conversion function and the first test data. The processor can be configured to calculate one or more of a test time savings or a computational complexity savings based on the difference between: (i) the conversion test time and the non-conversion test time, or (ii) the conversion computational complexity and the non-conversion computational complexity.

[0129] Using a transformation function, it can be classified to determine the ODD space. For example, when testing for different weather conditions, a sunny test scenario can be executed using sunny weather, and a rainy test scenario can be executed using rainy weather. The transformation function can be configured to transform one or more of the sunny test scenario or the rainy test scenario to collect data related to test scenarios for additional weather conditions (e.g., snowy, windy, dark, etc.).

[0130] The output of the learning correlation block 680 (e.g., the transformation function 690 between two data probability distributions) can be provided to the probability distribution block 570 (e.g., the annotated data probability distribution block 571) shown in FIG. 5 and transmitted to the extended test modality-specific data probability distribution 574. The output of the test modality-specific annotated data probability distribution (e.g., 573a, 573b, 573c) can be provided to the extended test modality-specific data probability distribution 574. The extended test modality-specific data probability distribution 574 can be transmitted to the intelligent job manager 540c.

[0131] FIG. 7 shows a process flow of an exemplary method 700 that can be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. The method 700 can be arranged according to at least one example described in the present disclosure.

[0132] The method 700 can be executed by processing logic that can include hardware (circuits, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine), or a combination of both, and this processing logic can be included in the processing device (e.g., a processor) 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0133] Method 700 may begin at block 705, where processing logic may identify a first parameter set defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints.

[0134] At block 710, processing logic may map the first parameter set to an internal parameter set using an internal parameter format that includes one or more internal axes, one or more internal relationships, and one or more internal constraints.

[0135] At block 715, processing logic may generate a test configuration object configured to link to one or more objects using the internal parameter set.

[0136] At block 720, processing logic may send the test configuration object to a test modality for testing.

[0137] The processor may further be configured to execute instructions to cause the device to identify a compile source of an internal parameter format for mapping an internal parameter set to a first parameter set, or to identify a metric of the internal parameter format after testing, or to format a test configuration object into a test modality format for a test modality, or to send the test configuration object to a test modality associated with the test modality, or in an internal parameter format converter plugin, to convert test modality format data into internal parameter format data, or in an internal parameter format converter plugin, to convert axis data of a time-based metric signal format into internal parameter format data, or to write the axis data to a database. The test modality may include one or more of a specific scenario language simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface. The first scenario format may include an open scenario format that may include an OSC definition language.

[0138] Modifications, additions, or omissions may be made to method 700 without departing from the scope of the present disclosure. For example, in some examples, method 700 may include any number of other components that may not be explicitly illustrated or described.

[0139] FIG. 8 shows a process flow of an exemplary method 800 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 800 may be arranged according to at least one example described in the present disclosure.

[0140] Method 800 may be executed by processing logic that may include hardware (such as circuitry, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0141] Method 800 may begin at block 805, where the processing logic may identify a first set of parameters defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationship, and one or more first scenario format constraints.

[0142] At block 810, the processing logic may map the one or more internal axes and the first set of parameters to an internal set of parameters that includes one or more internal relationships and one or more internal constraints.

[0143] At block 815, the processing logic may calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0144] The processor may be further configured to execute instructions for causing the device to select a set of axes, display a result based on a set of axes on a display device, select a subset of a set of axes, display a subset of a result based on a subset of a set of axes on a display device, retrieve one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric. The parametric metric may be based on a use case.

[0145] Modifications, additions, or omissions may be made to method 800 without departing from the scope of the present disclosure. For example, in some instances, method 800 may include any number of other components that may not be explicitly illustrated or described.

[0146] FIG. 9 shows a process flow of an exemplary method 900 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 900 may be arranged according to at least one example described in the present disclosure.

[0147] Method 900 may be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0148] Method 900 may begin at block 905 where the processing logic may receive scenario format data.

[0149] At block 910, the processing logic may convert the scenario format data into internal parameter format data.

[0150] At block 915, the processing logic may use the internal parameter format data to generate a test configuration object.

[0151] At block 920, the processing logic may use a test modality to test the test configuration object.

[0152] The processor is further configured to execute instructions to cause the device to perform one or more of: format test configuration objects in the device into a test modality format for a test modality by a specific scenario format simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface; associate the test configuration objects with a test modality related to the test modality and send the test configuration objects to the test modality; convert test modality format data to internal parameter format data in an internal parameter format converter plugin; convert axis data of a time-based metric signal format to internal parameter format data in the internal parameter format converter plugin; write axis data to a database; calculate parametric metrics based on use cases based on one or more of parametric performance metrics or parametric coverage metrics; extract one or more axis values for a test modality; calculate one or more axis-specific metrics based on the one or more axis values; and generate test configuration objects based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metrics.

[0153] Modifications, additions, or omissions may be made to method 900 without departing from the scope of the present disclosure. For example, in some instances, method 900 may include any number of other components that may not be explicitly illustrated or described.

[0154] FIG. 10 illustrates a process flow of an exemplary method 1000 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1000 may be arranged according to at least one example described in the present disclosure.

[0155] Method 1000 may be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0156] Method 1000 may start at block 1005, where the processing logic may test a first test sample batch that includes one or more first test samples and output one or more first test sample results.

[0157] At block 1010, the processing logic may identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed.

[0158] At block 1015, the processing logic may calculate a parametric metric that includes one or more of a parametric performance metric or a parametric coverage metric based on the one or more asynchronous test sample results.

[0159] At block 1020, the processing logic may adjust a second test sample batch based on the parametric metric.

[0160] The processor causes the device to determine one or more asynchronous test sample results based on real-world data, determine one or more asynchronous test sample results based on parametric metrics including one or more of parametric performance metrics or parametric coverage metrics, end a first sample batch based on parametric metrics, convert a plurality of first test samples from an internal parameter format to a test modality format, convert one or more asynchronous test sample results from a test modality format to an internal parameter format, and may be further configured to execute instructions to cause one or more of adjusting a second test sample batch based on test adjustment configuration parameters including one or more of a learning rate decay parameter, a sample size parameter, hyperparameters, an algorithm selection parameter, etc. The plurality of first test sample results may be generated using one or more different test modalities. Alternatively, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0161] Modifications, additions, or omissions may be made to method 1000 without departing from the scope of the present disclosure. For example, in some examples, method 1000 may include any number of other components that may not be explicitly illustrated or described.

[0162] FIG. 11 shows a process flow of an exemplary method 1100 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1100 may be arranged according to at least one example described in the present disclosure.

[0163] Method 1100 can be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0164] Method 1100 may start at block 1105, where the processing logic may identify one or more asynchronous test sample results of one or more first test samples before the first test sample batch completes the test.

[0165] In block 1110, the processing logic may calculate a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric.

[0166] In block 1115, the processing logic may select a set of algorithms based on the parametric metric.

[0167] In block 1120, the processing logic may apply the algorithms of the set of algorithms to a second test sample batch before the first test sample batch completes the test.

[0168] The processor causes the device to calculate a parametric metric that includes one or more of a confidence level, a confidence interval, a normalized metric, or a sample density, select a set of axes based on the parametric metric, display a result based on the set of axes on a display device, select a subset of the set of axes, display a subset of the results based on the subset of the set of axes on the display device, retrieve one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on the one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric, and may be further configured to execute instructions for causing the device to perform one or more of the foregoing. The parametric metric may be based on a use case. The parametric metric may be calculated using one or more of an objective function or real-world data. Alternatively, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0169] Modifications, additions, or omissions may be made to method 1100 without departing from the scope of the disclosure. For example, in some instances, method 1100 may include any number of other components that may not be explicitly illustrated or described.

[0170] FIG. 12 shows a process flow of an exemplary method 1200 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1200 may be arranged in accordance with at least one example described in the present disclosure.

[0171] Method 1200 may be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0172] Method 1200 may start at block 1205, where the processing logic may test a first test sample batch in a test modality format, and the first sample batch includes one or more first test samples for outputting one or more first test sample results.

[0173] At block 1210, the processing logic may identify one or more asynchronous test sample results of a plurality of the first test samples before the first test sample batch is completed.

[0174] At block 1215, the processing logic may convert one or more asynchronous test sample results from a test modality format to an internal parameter format.

[0175] At block 1220, the processing logic may adjust a second test sample batch based on one or more asynchronous test sample results.

[0176] The processor causes the device to format a second test sample batch into a test modality format for the test modality, or to send the second test sample batch to a test modality associated with the test modality, including one or more of a specific scenario format simulator, a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface, or to convert axis data in a time-based metric signal format into internal parameter format data in an internal parameter format converter plugin, or to write the axis data to a database, or to calculate a parametric metric based on a use case, based on one or more of a parametric performance metric or a parametric coverage metric, or to extract one or more axis values for the test modality, or to calculate one or more axis-specific metrics based on one or more axes and generate a second test sample batch based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric. Optionally, or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, etc.) is achieved.

[0177] Modifications, additions, or omissions may be made to method 1200 without departing from the scope of the present disclosure. For example, in some instances, method 1200 may include any number of other components that may not be explicitly illustrated or described.

[0178] FIG. 13 shows a process flow of an exemplary method 1300 that can be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1300 can be arranged according to at least one example described in the present disclosure.

[0179] Method 1300 can be executed by processing logic that can include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic can be included in the processing device 1702 of FIG. 17, or another device, a combination of devices, or a system.

[0180] Method 1300 can start at block 1305, where the processing logic can calculate an internal parameter set that includes one or more internal axes, one or more internal relationships, and one or more internal constraints.

[0181] At block 1310, the processing logic can calculate a metric that includes one or more of a performance metric or a coverage metric using the internal parameter set.

[0182] The processor can be further configured to execute instructions for causing the device to calculate a metric using one or more of a confidence level, a confidence interval, a normalized format, or a sample density, determine an operational design domain (ODD) coverage of the vehicle based on the metric, determine an operational design domain (ODD) performance of the vehicle based on the metric, query a safety characteristic of the vehicle, calculate a safety metric based on the metric in response to querying the safety characteristic of the vehicle, calculate an integrated metric based on a performance metric and a coverage metric, calculate an integrated metric based on one or more of an objective function or real-world data, or calculate a metric using parallelization to maximize one or more of an operational design domain (ODD) coverage or ODD performance.

[0183] Modifications, additions, or omissions can be made to method 1300 without departing from the scope of the present disclosure. For example, in some instances, method 1300 may include any number of other components that may not be explicitly illustrated or described.

[0184] FIG. 14 shows a process flow of an exemplary method 1400 that can be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1400 can be arranged according to at least one example described in the present disclosure.

[0185] Method 1400 can be executed by processing logic that can include hardware (such as circuits, dedicated logic, etc.), software (such as that executed on a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic can be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0186] Method 1400 can start at block 1405, where the processing logic can identify a first parameter set defined using a first scenario format that includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints.

[0187] At block 1410, the processing logic can map the one or more internal axes and the first parameter set to an internal parameter set that includes one or more internal relationships and one or more internal constraints.

[0188] At block 1415, the processing logic can calculate a metric based on one or more of a performance metric or a coverage metric.

[0189] The processor may be further configured to execute instructions to cause the device to select a set of axes, display a result based on the set of axes on a display device, select a subset of the set of axes, display a subset of the results based on the subset of the set of axes on the display device, retrieve one or more axis values for one or more test modalities, calculate one or more axis-specific metrics based on the one or more axis values, or generate a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on a metric. The metric may be based on a use case.

[0190] Modifications, additions, or omissions may be made to method 1400 without departing from the scope of the present disclosure. For example, in some instances, method 1400 may include any number of other components that may not be explicitly illustrated or described.

[0191] FIG. 15 illustrates a process flow of an exemplary method 1500 that may be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. Method 1500 may be arranged according to at least one example described in the present disclosure.

[0192] Method 1500 may be executed by processing logic that may include hardware (such as circuits, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine), or a combination of both, and this processing logic may be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0193] Method 1500 may begin at block 1505, where the processing logic may test a first test sample batch that includes one or more first test samples and output one or more first test sample results.

[0194] In block 1510, the processing logic may identify one or more asynchronous test sample results of a plurality of first test samples before the first test sample batch is completed.

[0195] In block 1515, the processing logic may calculate a metric including one or more of a performance metric or a coverage metric based on one or more asynchronous test sample results.

[0196] In block 1520, the processing logic may adjust a second test sample batch based on the metric.

[0197] The processor may cause the device to determine one or more asynchronous test sample results based on a parametric metric including one or more of a parametric performance metric or a parametric coverage metric, or determine one or more asynchronous test sample results based on real-world data, or end a first sample batch based on the metric, or convert a plurality of first test samples from an internal parameter format to a test modality format, or convert one or more asynchronous test sample results from a test modality format to an internal parameter format, or further configured to execute instructions to cause one or more of adjusting a second test sample batch based on a test adjustment configuration parameter including one or more of a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, etc. The first test sample results may be generated using one or more different test modalities.

[0198] Modifications, additions, or omissions may be made to method 1500 without departing from the scope of the present disclosure. For example, in some examples, method 1500 may include any number of other components that may not be explicitly illustrated or described.

[0199] FIG. 16 shows a process flow of an exemplary method 1600 that can be used for vehicle testing, simulation, or verification according to at least one example described in the present disclosure. The method 1600 can be arranged according to at least one example described in the present disclosure.

[0200] The method 1600 can be executed by processing logic that can include hardware (such as circuits, dedicated logic, etc.), software (such as that executed by a computer system or a dedicated machine, etc.), or a combination of both, and this processing logic can be included in the processing device 1702 of FIG. 17, or another device, combination of devices, or system.

[0201] The method 1600 can start at block 1605, where the processing logic can identify first test data based on a first test modality that includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data.

[0202] At block 1610, the processing logic can calculate a conversion function configured to convert the first test data from the first test modality to a second test modality different from the first test modality.

[0203] At block 1615, the processing logic can use the conversion function to calculate second test data that includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data based on the first test data.

[0204] The processor causes the device to calculate an information loss metric, which is the amount of information loss for the second test data compared to the first test data, based on one or more of the first test data or the second test data; calculate a risk metric, which is the amount of risk for the second test data compared to the risk for the first test data, based on one or more of the first test data or the second test data; calculate an uncertainty metric, which is the amount of uncertainty for the second test data compared to the uncertainty for the first test data, based on one or more of the first test data or the second test data; calculate a conversion test time for calculating the second test data based on the first test data using a conversion function; calculate a non-conversion test time for calculating the second test data without using the conversion function and the first test data; or calculate a test time savings based on the difference between the conversion test time and the non-conversion test time, and may be further configured to execute instructions for causing the device to perform one or more of the foregoing. The first test modality may be one or more of software-in-the-loop (SIL), hardware-in-the-loop (HIL), or track testing. The second test modality may be one or more of software-in-the-loop (SIL), hardware-in-the-loop (HIL), or track testing.

[0205] Modifications, additions, or omissions may be made to method 1600 without departing from the scope of the present disclosure. For example, in some instances, method 1600 may include any number of other components that may not be explicitly illustrated or described.

[0206] For simplicity of explanation, the methods and / or process flows described herein are depicted and described as a series of acts. However, the acts according to the present disclosure can be performed in various orders and / or simultaneously, and with other acts not presented and described herein. Further, not all of the acts shown may be used to implement the method according to the disclosed subject matter. Additionally, those skilled in the art will understand and recognize that the method can alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, the methods disclosed herein can be stored on a manufactured article, such as a non-transitory computer-readable medium, to facilitate transmission and transfer of such methods to a computing device. As used herein, the term manufactured article is intended to encompass a computer program accessible from any computer-readable device or storage medium. Although shown as individual blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

[0207] In some examples, a vehicle (e.g., an AV) simulation, test, and verification system may include an environment representation creation module that includes code and routines configured to enable a computing device to perform one or more operations related to generating a 3D environment representation, preparing and generating scenarios, simulating scenarios, and verifying and / or evaluating the simulation of scenarios. Additionally, or alternatively, the environment representation creation module may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling the performance of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other instances, the environment representation creation module may be implemented using a combination of hardware and software. In the present disclosure, the operations described as being performed by the environment representation creation module may include operations that may direct the system corresponding to the environment representation creation module to perform.

[0208] In some examples, the environment representation creation module may be configured to generate a 3D environment representation. The environment representation creation module may generate a 3D environment representation using any suitable 3D modeling technique. In some examples, the environment representation creation module may use map data as input data in generating the 3D environment representation. For example, the 3D environment of the 3D environment representation may represent a geographic area represented by the map data.

[0209] In some examples, the 3D environment representation may include 3D models of one or more objects in a geographic area as described by the map data. For example, the 3D environment representation may include a complete 3D model of a simulated driving environment.

[0210] A vehicle (e.g., AV) simulation, test, and verification system may include a machine learning circuit and / or software for more efficiently and preventively identifying bad cases. A vehicle (e.g., AV) simulation, test, and verification system may include a circuit and / or software for pruning, including parameter-based pruning. A vehicle (e.g., AV) simulation, test, and verification system may include a circuit and / or software for implementing scenarios, runtime constraints for simulations, or runtime constraints within a 3D environment representation.

[0211] FIG. 17 shows a diagrammatic representation of a machine in an exemplary form of a computing device 1700 in which a set of instructions, which when executed cause the machine to perform any one or more of the methods discussed herein, may be executed. The computing device 1700 may include, for example, a rack-mounted server, a router computer, a server computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, or any computing device having at least one processor, within which a set of instructions for causing the machine to perform any one or more of the methods discussed herein may be executed. In an alternative example, the machine may be connected (e.g., network-connected) to other machines in a local area network (LAN), intranet, extranet, or the Internet. The machine may operate in the capacity of a server machine in a client-server network environment. Further, although only a single machine is shown, the term "machine" may also include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0212] Exemplary computing device 1700 includes a processing device (e.g., a processor) 1702, a main memory 1704 (e.g., dynamic random access memory (DRAM) such as read-only memory (ROM), flash memory, synchronous DRAM (SDRAM)), a static memory 1706 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1716, which communicate with each other via a bus 1708.

[0213] Processing device 1702 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. More specifically, processing device 1702 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set, or a processor implementing a combination of instruction sets. Processing device 1702 may also include one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processing device 1702 is configured to execute instructions 1726 for performing the operations and steps considered herein.

[0214] Computing device 1700 may further include a network interface device 1722 that can communicate with network 1718. Computing device 1700 may also include a display device 1710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1712 (e.g., a keyboard), a cursor control device 1714 (e.g., a mouse), and a signal generation device 1720 (e.g., a speaker). In at least one example, display device 1710, alphanumeric input device 1712, and cursor control device 1714 may be combined into a single component or device (e.g., an LCD touch screen).

[0215] The data storage device 1716 may include a computer-readable storage medium 1724 storing one or more sets of instructions 1726 that implement any one or more of the methods or functions described herein. The instructions 1726 may also be fully or at least partially present within the main memory 1704 and / or within the processing device 1702 during execution by the computing device 1700, and the main memory 1704 and the processing device 1702 also constitute a computer-readable medium. The instructions may be further transmitted or received via the network interface device 1722 over the network 1718.

[0216] The computer-readable storage medium 1724 is shown as a single medium in one example, but the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) storing one or more sets of instructions. The term "computer-readable storage medium" may also include any medium capable of storing, encoding, or carrying a set of instructions for machine execution and causing the machine to execute any one or more of the methods of the present disclosure. Thus, the term "computer-readable storage medium" may include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0217] (Example) The following provides examples of the present disclosure.

[0218] Example 1: Lane support using an open scenario In this example, the target vehicle and the host vehicle are described with respect to the lane. The host vehicle travels along a route and is initially in the same lane as the target vehicle. The target vehicle is accelerating towards the host vehicle and has a distance of approximately 0.1 meter to the right lane demarcation line. At the start of the scenario, the host vehicle has a distance of approximately 100 meters between itself and the target vehicle and a distance of approximately 0.5 to the right lane demarcation line. During the scenario, the host vehicle moves to a different lane. The host vehicle returns to the same lane as the target vehicle at the end of the scenario.

[0219] An example of the OpenSCENARIO 2.0 scenario format is provided in Table I. [Table 1]

[0220] Example 2: Conversion from OpenSCENARIO to Internal Parameter Format The OpenSCENARIO of the scenario description language can be converted to the internal parameter format.

[0221] The OpenSCENARIO code provided in Table I includes the following: (i) axes (e.g., vehicle, route, lane, lateral, position, speed, etc.), (ii) relationships (e.g., target vehicle and host vehicle within the same lane), and (iii) constraints (e.g., at least two lanes). The OpenSCENARIO code can be mapped to the internal parameter format by identifying the OpenSCENARIO axes, relationships, and constraints and converting those axes, relationships, and constraints to the internal parameter format (e.g., intermediate YAML format).

[0222] Example 3: Parallel Tests Using Asynchronous Information Two different scenarios of the OpenSCENARIO language can be converted into an internal parameter format. Both scenarios can be tested in the same batch as the first test and the second test. The first test based on the first scenario can complete the test before the second test based on the scenario completes. The result of the first test based on the first scenario can be used in the third test based on the second scenario.

[0223] Provide the first scenario in Table I and the second scenario in Table II.

Table 2

[0224] In the second scenario, the host vehicle does not change lanes in response to the target vehicle accelerating behind it in the same lane. The host vehicle accelerates from 100 km / h to 130 km / h. The target vehicle starts the scenario at a speed of 150 km / h. This scenario can be used to test the response of the host vehicle to an approaching vehicle.

[0225] The first test results from the first scenario may indicate that if the host vehicle does not change lanes, there is a possibility that the host vehicle will collide with the target vehicle from behind. If these first test results are completed before the completion of the second test results, they can be used to adjust the third test based on the second scenario. Using the knowledge that the host vehicle cannot change lanes and the target vehicle will collide with the host vehicle, for example, by reducing the speed of the target vehicle to test whether the host vehicle can sense the target vehicle and avoid the collision, the third scenario can be modified.

[0226] Example 4: Parametric Metric Using Internal Parameter Format and Asynchronous Information Using the scenarios detailed in Table I ("post-run") and II ("pre-run"), parametric metrics can be calculated. Converting the scenarios to an internal parameter format (e.g., an intermediate YAML format) can enhance the performance and information gain received for parametric metrics (e.g., one or more of parametric performance metrics or parametric coverage metrics).

[0227] Convert the scenarios represented using OpenSCENARIO to an internal parameter format to facilitate conversion of the scenarios to different test modalities (e.g., SIL, HIL, track), and the test results can be converted back from different test modalities to the internal parameter format. Metrics from different test modalities can be compared using the same language. Therefore, parametric coverage metrics and parametric performance metrics can be calculated with higher accuracy and precision compared to calculations and comparisons based on different languages.

[0228] For the scenarios described in post-run and pre-run, one performance metric can be the frequency at which the host vehicle collides as a result of: (i) changing lanes so that the target vehicle can pass, or (ii) staying in the same lane and increasing speed. The coverage metric can be calculated based on the number of different situations (e.g., different weather, vehicle speed, road type, etc.) related to these scenarios in post-run and pre-run that are tested based on the test data from the post-run and pre-run scenarios.

[0229] Converting the OpenSCENARIO format to an internal parameter format (e.g., an intermediate YAML format) can facilitate combining data from the OpenSCENARIO format with data from a different scenario language (i.e., not the OpenSCENARIO language) and calculating metrics based on the combined data. Different scenario languages can collect data related to weather conditions and road types for the ego vehicle and target vehicle in different scenarios (e.g., a composite lane change scenario where the ego vehicle and the target vehicle switch lanes almost simultaneously while driving within a selected distance from each other). Data from this scenario can be converted to an internal parameter format (e.g., an intermediate YAML format) and combined with data from the scenarios described later and earlier. The sum of the coverage metrics provided by later and earlier and the coverage metric provided by the composite lane change scenario may not be the same as the coverage metric when the scenarios described in Tables I and II are combined with the coverage metric before calculating the coverage metric.

[0230] Using asynchronous test results can increase information gain and reduce computational complexity and test time. For situations where the later scenario and the composite lane change scenario complete the test before the earlier scenario completes the test, the later scenario and the composite lane change scenario can be converted to an internal parameter format (e.g., an intermediate YAML format) before the performance and / or coverage metric is calculated. The resulting performance and / or coverage metric can be used to select a specific algorithm to be applied to subsequent test batches before the test data from the earlier scenario is complete.

[0231] Example 5: Metrics Using Internal Parameter Format and Asynchronous Information Rearward, forward, and combined lane change scenarios can be used with a metric. This metric can be calculated using one or more of a confidence level, a confidence interval, a normalized format, or a sample density. For example, a rearward scenario can have a 2% collision rate, a forward scenario can have a 3% collision rate, and a combined lane change scenario can have a 5% collision rate. Confidence levels and intervals can be calculated for these scenarios. The 2% collision rate for the rearward scenario can have a confidence interval of 2.5% - 3.5% with a 95% confidence level. The 3% collision rate for the forward scenario can have a confidence interval of 3.5% - 4.5% with a 97% confidence level. The 5% collision rate for the combined lane change scenario can have a confidence interval of 3% - 8% with an 85% confidence level.

[0232] The normalized format can be calculated by combining three different scenarios using an internal parameter format and normalizing the data such that the area under the curve is approximately equal to 1. The sample density can be calculated based on the sample density for each of the scenarios.

[0233] The metric can be combined into a single metric (e.g., an integrated metric) that combines a coverage metric and a performance metric. A single metric can be used to determine what to test. For example, a coverage metric based on rearward, forward, and combined lane change scenarios can indicate that in a blizzard, there is no coverage for a scenario where two vehicles simultaneously change lanes into the same lane and travel at a speed of 300 km / h, but the single metric can calculate that the scenario will not be tested because the performance (e.g., collision) is known.

[0234] Example 6: Transformation Function The first test data calculated using the first test modality can be converted into second test data for the second test modality. For example, the first test data can be SIL test data, and the second test data can be HIL data. A suitable machine learning technique, such as supervised machine learning (e.g., regression) using training data for the first test modality and the second test modality, can be used to determine the conversion function between the first test modality and the second test modality. The objective function can be optimized to generate one or more functions used to convert between test modalities.

[0235] For rearward, forward, and composite lane change scenarios, the test modality can include SIL test data. The SIL test data can be converted into HIL test data using the conversion function. Converting the SIL test data into HIL test data using the conversion function can facilitate shortening the test time.

[0236] In some examples, the different components, modules, engines, and services described herein can be implemented as objects or processes (e.g., as separate threads) running on a computing system. Although some of the systems and methods described herein are generally described as being implemented in software (stored in and / or executed by hardware), a specific hardware implementation, or a combination of software and a specific hardware implementation is also possible and contemplated.

[0237] As used herein, and especially in the appended claims (e.g., the body of the appended claims), the terms are generally intended to be "open" terms (e.g., the term "comprising" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including but not limited to," etc.).

[0238] In addition, when a particular number of the introduced claim limitations is intended, such intention shall be expressly recited in the claim, and in the absence of such recitation, such intention does not exist. For example, by way of illustration, the appended claims below may include the use of the preambles “at least one” and “one or more” for introducing claim limitations. However, the use of such phrases shall not be construed as implying that the introduction of a claim limitation by an indefinite article such as “a” or “an” limits any particular claim containing such introduced claim limitation to examples containing only one such limitation, and the same shall apply to the use of definite articles used to introduce claim limitations.

[0239] In addition, it is understood that even if a particular number of the introduced claim limitations is expressly recited, such recitation shall be construed to mean at least the recited number (e.g., a recitation of “two limitations” without other modifiers means at least two limitations, or two or more limitations). Further, in those instances where conventions similar to “at least one of A, B, and C” or “one or more of A, B, and C” are used, generally such constructions are intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together. For example, the use of the term “and / or” is intended to be construed in this manner.

[0240] Furthermore, regardless of the description, claims, or drawings, any disjunctive word or phrase presenting two or more alternative terms should be understood as contemplating the possibility of including one of the terms, any of the terms, or both terms. For example, the phrase "A or B" should be understood as including the possibilities of "A" or "B" or "A and B".

[0241] In addition, the use of terms such as "first", "second", "third", etc. is not necessarily used herein to mean a particular order or number of elements. Generally, terms such as "first", "second", "third", etc. are used to distinguish different elements as general identifiers. If it is not indicated that terms such as "first", "second", "third", etc. mean a particular order, these terms should not be understood to mean a particular order. Furthermore, if it is not indicated that terms such as "first", "second", "third", etc. mean a particular number of elements, these terms should not be understood to mean a particular number of elements. For example, a first something device may be described as having a first surface, and a second something device may be described as having a second surface. The use of the term "second surface" with respect to the second something device may be to distinguish such a surface of the second something device from the "first surface" of the first something device, and does not mean that the second something device has two surfaces.

[0242] All examples and conditional language recited herein are intended for the educational purpose of assisting the reader in understanding the concepts contributed by the inventors to facilitate the present invention and the technology, and should be construed as not being limited to such specifically recited examples and conditions. Although the examples of the present disclosure have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure.

Claims

1. A device for vehicle simulation, a memory, a processor operably coupled to the memory, the device being caused by the processor to identify a first parameter set defined using a first scenario format, the first parameter set including one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints, map the first parameter set to an internal parameter set using an internal parameter format including one or more internal axes, one or more internal relationships, and one or more internal constraints, generate a test configuration object configured to link to a plurality of objects using the internal parameter set, and execute instructions for causing the test configuration object to be transmitted to a test modality for testing, the device comprising the processor.

2. The device according to claim 1, wherein the processor is further configured to execute instructions for causing the device to identify a compilation source of the internal parameter format for mapping the internal parameter set to the first parameter set.

3. The device according to claim 1, wherein the processor is further configured to execute instructions for causing the device to identify a metric of the internal parameter format after testing.

4. The device according to claim 1, wherein the test modality includes one or more of a software-in-loop (SIL) simulator plugin interface, a hardware-in-loop (HIL) simulator plugin interface, or a track test plugin interface.

5. The device according to claim 1, wherein the processor is further configured to execute instructions for causing the device to format the test configuration object into a test modality format for the test modality and transmit the test configuration object to the test modality.

6. The device according to claim 1, wherein the first scenario format includes one or more of an Open Scenario (OSC) definition language.

7. ​ ​ ​ ​ The processor causes the device to The device according to claim 1, further configured to execute instructions for causing the internal parameter format converter plugin to convert test modality format data into internal parameter format data. **Claim 8** The processor causes the device to In the internal parameter format converter plugin, convert the axis data of the time-based metric signal format into internal parameter format data, The device according to claim 1, further configured to execute instructions for causing the axis data to be written to a database. **Claim 9** A device for vehicle testing, comprising: A memory; A processor operably coupled to the memory, the processor causing the device to Identify a first parameter set defined using a first scenario format, the first parameter set including One or more first scenario format axes, One or more first scenario format relationships, and One or more first scenario format constraints; Map the first parameter set to an internal parameter set including One or more internal axes, One or more internal relationships, and One or more internal constraints; and Execute instructions for calculating a parametric metric based on one or more of a parametric performance metric or a parametric coverage metric. **Claim 10** The device according to claim 9, wherein the parametric metric is based on a use case. **Claim 11** The processor causes the device to Select a set of axes based on the parametric metric, or Display a result based on the set of axes on a display device, or Select a subset of the set of axes, or The device according to claim 9, further configured to execute instructions for displaying a subset of the result based on the subset of the set of axes on the display device. **Claim 12** The processor causes the device to Retrieve one or more axis values for one or more test modalities The device according to claim 9, further configured to execute instructions for calculating one or more axis-specific metrics based on the one or more axis values. **Claim 13** The processor causes the device to The device according to claim 9, further configured to execute instructions for generating a test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric. **Claim 14** A non-transitory computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, cause a vehicle simulator to receive scenario format data, convert the scenario format data into internal parameter format data, generate a test configuration object using the internal parameter format data, and test the test configuration object using a test modality. **Claim 15** When the instructions are executed by the one or more processors, the vehicle simulator is further caused to format the test configuration object into a test modality format for the test modality, send the test configuration object to the test modality, The non-transitory computer-readable storage medium according to claim 14, wherein the test modality includes one or more of a software-in-the-loop (SIL) simulator plugin interface, a hardware-in-the-loop (HIL) simulator plugin interface, or a track test plugin interface. **Claim 16** When the instructions are executed by the one or more processors, the vehicle simulator is further caused to convert test modality format data into internal parameter format data in an internal parameter format converter plugin. **Claim 17** When the instructions are executed by the one or more processors, the vehicle simulator is further caused to In the internal parameter format converter plugin, convert the axis data of the time-based metric signal format into internal parameter format data, The non-transitory computer-readable storage medium according to claim 14, which causes the database to write the axis data. **Claim 18** When the instructions are executed by the one or more processors, further cause the vehicle simulator to Calculate parametric metrics based on use cases based on one or more of parametric performance metrics or parametric coverage metrics, the non-transitory computer-readable storage medium according to claim 14. **Claim 19** When the instructions are executed by the one or more processors, further cause the vehicle simulator to Extract one or more axis values for the test modality, Calculate one or more axis-specific metrics based on the one or more axis values, the non-transitory computer-readable storage medium according to claim 14. **Claim 20** When the instructions are executed by the one or more processors, further cause the vehicle simulator to Generate the test configuration object based on one or more axis-specific metrics based on one or more axis values selected based on the parametric metric, the non-transitory computer-readable storage medium according to claim 18.

Images