Scene-independent parametric spatial interior representation

Abstract

The invention relates to an apparatus for vehicle simulation, testing and verification. The apparatus may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first set of parameters defined using a first scene format, wherein the first set of parameters includes one or more first scene format axes, one or more first scene format relationships, and one or more first scene format constraints. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object using the internal parameter set, and send the test configuration object to a test modality for testing.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 366,770, filed on June 21, 2022, and U.S. Provisional Application No. 63 / 366,830, filed on June 22, 2022, the disclosures of which are incorporated herein by reference in their entireties.

[0003] The present disclosure relates to vehicle simulation, testing and validation systems and methods. Background Art

[0004] Unless otherwise indicated herein, the materials described herein are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0005] Simulation environments can be used to test software. For example, a simulated driving environment can be used to test software for an autonomous vehicle. An autonomous vehicle can use sensors to perceive its environment. The control system can model the sensory input from the sensors to determine a navigation path and make decisions in response to traffic controls such as red lights, roundabouts, stop signs, speed limit changes, and other vehicles.

[0006] Vehicles may have different levels of automation. For some vehicles, the automated system may provide warnings but may not otherwise control the vehicle. For other vehicles, the vehicle may operate without human intervention in many different surfaces, seasons, and weather conditions. Therefore, apparatus, systems, and methods for testing, simulating, and validating vehicles with multiple levels of automation may be useful.

[0007] The subject matter claimed in this disclosure is not limited to examples that overcome any disadvantages or that operate only in environments such as those described above. Rather, this background is provided merely to illustrate one example technology area in which some examples described in this disclosure may be practiced. Summary of the invention

[0008] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first parameter set defined using a first scenario format, wherein the first parameter set includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object using the internal parameter set, wherein the test configuration object is configured to link to one or more objects. The processor may be configured to send the test configuration object to a test modality for testing.

[0009] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first parameter set defined using a first scenario format, wherein the first parameter set includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first parameter set to an internal parameter set, wherein the internal parameter set includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a parameter metric, wherein the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric.

[0010] A non-transitory computer-readable storage medium containing computer-executable instructions that, when executed by one or more processors, cause a vehicle simulator to: receive scenario format data; convert the scenario format data into internal parameter format data; generate a test configuration object using the internal parameter format data; and test the test configuration object using a test modality.

[0011] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples to output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results for the plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a parameter metric based on the one or more asynchronous test sample results, wherein the parameter metric includes one or more of a parameter performance metric or a parameter coverage metric. The processor may be configured to adjust a second test sample batch based on the parameter metric.

[0012] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify one or more asynchronous test sample results for one or more first test samples before a first test sample batch has completed testing. The processor may be configured to calculate a parameter metric, wherein the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric. The processor may be configured to select an algorithm set based on the parameter metric. The processor may be configured to apply an algorithm in the algorithm set to a second test sample batch before the first test sample batch has completed testing.

[0013] A computer-readable storage medium containing computer-executable instructions that, when executed by one or more processors, cause a vehicle tester to: test a first test sample batch in a test modality format, wherein the first sample batch includes one or more first test samples to output one or more first test sample results; identify one or more asynchronous test sample results for a plurality of first test samples before the first test sample batch is completed; convert the one or more asynchronous test sample results from the test modality format to an internal parameter format; and adjust a second test sample batch based on the one or more asynchronous test sample results.

[0014] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric using the internal parameter set, wherein the metric includes one or more of a performance metric or a coverage metric.

[0015] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first parameter set defined using a first scenario format, wherein the first parameter set includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first parameter set to an internal parameter set, wherein the internal parameter set includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric, wherein the metric is based on one or more of a performance metric or a coverage metric.

[0016] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples to output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results for the plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a metric based on the one or more asynchronous test sample results, wherein the metric includes one or more of a performance metric or a coverage metric. The processor may be configured to adjust a second test sample batch based on the metric.

[0017] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify first test data based on a first test modality, wherein the first test data includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data. The processor may be configured to calculate a transformation function, wherein the transformation function is configured to transform the first test data from the first test modality to a second test modality, wherein the second test modality is different from the first test modality. The processor may be configured to calculate second test data based on the first test data using the transformation function, wherein the second test data includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data.

[0018] The objects and advantages of the examples will be realized and attained by at least the elements, features, and combinations particularly pointed out in the claims.

[0019] Both the foregoing general description and the following detailed description are given by way of example and are explanatory and not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The examples will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0021] Figure 1A An example block diagram for vehicle simulation, testing, and validation is shown.

[0022] Figure 1B An example block diagram for vehicle simulation, testing, and validation is shown.

[0023] Figure 2 An example process flow for an internal parameter format used in vehicle simulation, testing, and validation is shown.

[0024] Figure 3AAn example process flow for parallelization in vehicle simulation, testing, and validation is shown.

[0025] Figure 3B An example process flow for parallelization in vehicle simulation, testing, and validation is shown.

[0026] Figure 4A An example process flow for metrics in vehicle simulation, testing, and validation is shown.

[0027] Figure 4B An example process flow for data indexing in vehicle simulation, testing, and validation is shown.

[0028] Figure 4C An example process flow for characterization in vehicle simulation, testing, and validation is shown.

[0029] Figure 4D An example process flow for probability distribution of data points in vehicle simulation, testing, and validation is shown.

[0030] Figure 5 An example process flow for probability distribution of data in vehicle simulation, testing, and validation is shown.

[0031] Figure 6 An example process flow for learning correlations in vehicle simulation, testing, and validation is shown.

[0032] Figure 7 An example process flow for an internal parameter format used in vehicle simulation, testing, and validation is shown.

[0033] Figure 8 An example process flow for an internal parameter format used in vehicle simulation, testing, and validation is shown.

[0034] Fig. 9 An example process flow for an internal parameter format used in vehicle simulation, testing, and validation is shown.

[0035] Fig.10 An example process flow for asynchronous messaging in vehicle simulation, testing, and validation is shown.

[0036] Fig.11 An example process flow for asynchronous messaging in vehicle simulation, testing, and validation is shown.

[0037] Fig.12 An example process flow for asynchronous messaging in vehicle simulation, testing, and validation is shown.

[0038] Fig.13 An example process flow for metrics in vehicle simulation, testing, and validation is shown.

[0039] Fig.14 An example process flow for metrics in vehicle simulation, testing, and validation is shown.

[0040] Fig.15 An example process flow for metrics in vehicle simulation, testing, and validation is shown.

[0041] Fig.16 An example process flow for a transformation function used in vehicle simulation, testing, and validation is shown.

[0042] Fig.17 A diagrammatic representation of a machine in the example form of a computing device within which a set of instructions may be executed, for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0043] A vehicle software test bench, which may be an autonomous vehicle software test bench, may be used to test vehicle software (or autonomous vehicle software). The vehicle software test bench may simulate a driving environment to test the vehicle software (e.g., to determine how the vehicle software will perform in various driving situations). For example, if the vehicle software is running on a vehicle driving within an actual geographic environment, the vehicle software test bench may simulate a three-dimensional (3D) geographic environment and provide inputs to the vehicle software that may be similar to inputs that the vehicle software may receive under real-world driving conditions.

[0044] Determining a set of scenarios to generate and run in simulation can be used to effectively test and validate vehicle systems, and more specifically, autonomous vehicle (AV) systems. Identifying a set of scenarios that adequately covers the test space to a statistical confidence level helps demonstrate the accuracy and safety of the underlying system under test.

[0045] Previous approaches to scenario generation and coverage calculation are based on three basic assumptions: (1) a scenario definition language optimized for simple (but not necessarily comprehensive) parameterization; (2) contextualization and introspection of simulation environments and stacks; and (3) access to closed-loop test environments. Based on these three assumptions, verification engineers can define the parameters that make up their tests, the failure conditions for the tests, and the range and step size of possible values ​​for each parameter. Combinations of parameters are then tested iteratively, and new combinations are selected to collect additional information about the stack. The calculation of coverage is then based on what parameter values ​​were tested in the test set. Simple calculations of coverage are performed by grouping parameters into multidimensional histograms and clusters.

[0046] A limitation of these previous approaches is their reliance on the details of each of the above three assumptions. Specifically, many previous jobs depend on proprietary scenario definition formats that can limit the expressiveness and extensibility of many aspects of the scenarios compared to using industry standard scenario languages. In addition, these scenario languages ​​are often closely associated with the ability to introspect and modify these scenarios. Moreover, the closed-loop simulation environment used for their execution may not be realistic enough to provide accurate and realistic prior information for adaptive sampling. Based on this, many of these algorithms rely entirely on previous samples that have been collected before selecting the next sample to be run. Therefore, the ability of previous methods to scale has been limited, making these methods primarily limited to research scenarios. Finally, the original method of calculating coverage results has limited the ability to evaluate the performance of the stack and test coverage relative to the importance of different scenarios.

[0047] Various aspects of the present disclosure provide systems and methods for enhancing simulation, scenario selection and environment generation, and other aspects. For example, the systems and methods provide enhanced efficiency, such as through information sharing between different test modalities. Increased information sharing allows for more logical and efficient sampling.

[0048] Additional efficiency gains can result from evaluation of the stack (e.g., performance and coverage), which may otherwise be a manual process, and the statistical interface can speed up the evaluation. Performance gains resulting from parallelizable simulations and different test modalities can result in sampling algorithms that can converge more quickly. And, using the disclosed techniques, AV testing can identify failure modes more quickly and can statistically validate autonomous stacks.

[0049] Other aspects may provide a set of meta-statistics that quantifies the amount of information loss that occurs when switching between different test modalities and risky / uncertain regions based on this transformation.The set of meta-statistics may quantify the amount of testing preserved in each modality based on this correlation.

[0050] In yet another aspect, the present disclosure includes methods for correlating information from software-in-the-loop (SIL), hardware-in-the-loop (HIL), and trace testing to make predictions about performance under different scenarios. Aspects may provide definitions of transformation functions between SIL, HIL, and trace test performance, coverage or metric distributions, and corresponding uncertainties.

[0051] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify a first parameter set defined using a first scenario format, wherein the first parameter set includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to generate a test configuration object using the internal parameter set, wherein the test configuration object is configured to link to one or more objects. The processor may be configured to send the test configuration object to a test modality for testing.

[0052] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to test a first test sample batch including one or more first test samples to output one or more first test sample results. The processor may be configured to identify one or more asynchronous test sample results for the plurality of first test samples before the first test sample batch is completed. The processor may be configured to calculate a parameter metric based on the one or more asynchronous test sample results, wherein the parameter metric includes one or more of a parameter performance metric or a parameter coverage metric. The processor may be configured to adjust a second test sample batch based on the parameter metric.

[0053] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to calculate an internal parameter set including one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to calculate a metric using the internal parameter set, wherein the metric includes one or more of a performance metric or a coverage metric.

[0054] An apparatus for vehicle simulation, testing, and validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to identify first test data based on a first test modality, wherein the first test data includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data. The processor may be configured to calculate a transformation function, wherein the transformation function is configured to transform the first test data from the first test modality to a second test modality, wherein the second test modality is different from the first test modality. The processor may be configured to calculate second test data based on the first test data using the transformation function, wherein the second test data includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data.

[0055] Embodiments of the present disclosure will be explained with reference to the drawings.

[0056] The scenario-independent parameter space internal representation can be used to map open and / or custom representations to the internal representation. The parameter space itself can be manipulated to reduce the complexity and difficulty of the problem of the search / optimization algorithm. Some example data representations can include the scenario definition language (Association for Standardization of Automation and Measurement Systems (ASAM) Open (OSC) 1.0, OSC 2.0 or any other scenario language), axes (variables and their possible values, and output test metrics), relationships (defining how variables relate to each other), constraints (defining the boundaries of axes and limits of relationships), or the like.

[0057] like Figure 1A As shown in FIG. 1 , a block diagram 100a for vehicle testing, simulation and validation is provided. A parameter set may be defined to include a mapping from any scenario language (e.g., an open scenario format such as OSC) or a dedicated scenario format such as a test specification for a HIL device (rig) to an internal parameter format. That is, various scenario definition languages ​​including SIL / HIL provided in the form of OSC 2.0 as shown in block 105a, SIL / HIL provided in the form of OSC 1.x as shown in block 105b, or traces (test cases) as shown in block 105c may be mapped to a parameter space representation that includes one or more constraints, one or more values, or one or more relationships, as shown in block 125.

[0058] Parallelization and / or asynchronous evaluation may provide maximum efficiency in information gain. Parallelized simulation may allow for large-scale collection of information and generation of subsequent samples. Asynchronous evaluation may allow for data collection from hardware-in-the-loop (HIL) / in-vehicle test results to continue to inform testing. In some aspects, techniques may include methods for using asynchronous information to make decisions about further optimization.

[0059] Thus, simulations can be performed in parallel. The system can exploit the parallelization assumption to enable asynchronous updates of the sampling algorithm and priors of the parameter space. Specifically, the system can have the ability to enqueue new scenarios, and can build updated priors based on all data collected so far, and use the priors to enqueue the next scenario. Additionally, the unified parameter space representation allows the system to also asynchronously update this parameter space performance using real-world data.

[0060] The parameter space representation as shown in block 125 may be used to generate a specific scenario as shown in block 135. The specific scenario may be input to a parallelized test operation including various test modalities (e.g., SIL, HIL, trace), as shown in block 145a. Results from the parallelized test may be collected for the different test modalities (e.g., SIL, HIL, trace), as shown in block 145b. The results shown in block 145b may be fed back to the parameter space representation as shown in block 125 to continue the process of generating other specific scenarios, other parallelized tests, etc.

[0061] Various metrics can be used to evaluate coverage and performance for scenario spaces. These scenarios can also allow security to be queried and audited in a quantifiable manner. Moreover, these aspects can enable sampling algorithms to optimize parallelization in a way that maximizes statistical information metrics. In addition, a single high-level metric with statistical significance can be used to describe performance and coverage. The metrics can include: (a) performance estimates with confidence levels and confidence intervals, where the uniform parameter space can be normalized; (b) coverage estimates based on sample density, where the uniform parameter space can be normalized; and (c) unified metrics on a uniform parameter space and a set of objective functions (e.g., Boolean or continuous functions) and real-world observations.

[0062] Thus, the results shown in block 145b may be input to an evaluation metric operation, as shown in block 155. The evaluation metric operation may be configured to transform the test results using the parameter space representation to generate performance / coverage metrics as described.

[0063] Information can be correlated between SIL, HIL, and trace tests to make inferences about performance in different situations. A transformation function can be determined between SIL, HIL, and trace test performance, coverage or metric distribution, and corresponding uncertainty. A set of meta-statistics can quantify the amount of information lost when transforming between different test modalities and risky / uncertain regions based on this transformation. A set of meta-statistics can quantify the amount of testing preserved in each modality based on this correlation.

[0064] exist Figure 1B An example process flow 100b for vehicle simulation, testing, and validation is provided in . A scenario cross-compiler 110 may be configured to map from a scenario language (e.g., an open scenario format such as OSC or a dedicated scenario format such as a test specification for a HIL device) to an internal parameter format (e.g., an intermediate YAML format). The internal parameter format may be provided to a sampling service 120, which may be configured to store and / or process one or more axes, one or more relations, or one or more constraints calculated using the internal parameter format. A linker service 130 may use the internal parameter format to create a test configuration object that may be linked between different objects, test results, or the like.

[0065] The linker service 130 may be configured to provide a test configuration object to a block 140, which may include one or more of a queue service, a parallel job scheduler, or an intelligent job manager. The queue service may submit the test configuration object to a plug-in or service based on satisfaction of a particular constraint. The parallel job scheduler may schedule jobs for different test modalities, simulators, or the like. The intelligent job manager may determine different jobs to be tested based on sampling of a probability distribution.

[0066] The test results generated by the combined operation of the queue service, parallel job scheduler, and intelligent job manager can be provided to block 150, which includes post-processing services, sampling services, data indexing, and characterization. Post-processing services can slow down the software. Sampling services, data indexing, and characterization can group together primary indexes with filtered values ​​based on feature vectors. Data from block 150 can be provided to block 160, which includes inference services and distribution post-processing. Inference services can filter data that is not related to the relevant test into a multidimensional distribution. Distribution post-processing can generate annotated data point probability distributions using data point probability distributions.

[0067] The annotated data point probability distribution may be provided to block 170, which may include various other probability distributions (augmented test modality specific data point probability distribution, test modality specific annotated data probability distribution). The probability distribution from block 170 may be provided to a learn correlation block 180, which may be configured to generate a transformation function 190. The transformation function 190 may be configured to transform between different test modalities or other distributions. The augmented test modality specific data point probability distribution may be provided to block 140 to continue the testing process.

[0068] Internal Representation

[0069] A variety of industry standard scenario formats or custom formats may be used for simulation, testing, and validation of vehicles such as autonomous vehicles. A scenario-independent internal representation of a parameter space (e.g., an internal parameter format) and mapping between an open standard and this internal representation (e.g., an internal parameter format) may allow mapping between various industry standard scenario formats or custom formats and this internal representation (e.g., an internal parameter format). The parameter space internal representation (e.g., an internal parameter format) may allow for the inclusion of information across one or more test modalities to facilitate additional information about the coverage and performance of the parameter space.

[0070] The internal representation of the parameter space (e.g., an internal parameter format) may include one or more components, comprising: (i) one or more axes, which may include one or more variables and possible values ​​for the one or more variables; (ii) one or more relationships, which may include one or more relationships between one or more variables of the one or more axes, and one or more combinations between one or more variables of the one or more axes; and (iii) one or more constraints, which may: define one or more limits of the variable relative to other variables of the one or more variables, and define other areas in the scene space that may be invalid.

[0071] Using this internal representation of the parameter space (e.g., an internal parameter format), various statistical operations can be performed in a manner that is independent of the scene and test modality. Specifically, various statistical operations, such as multidimensional clustering, nearest neighbor analysis, parametric (or non-parametric) and / or linear (or non-linear) regression, or the like, can be performed to more efficiently inform the sampling algorithm compared to a baseline sampling algorithm in which the various statistical operations are not used. Specifically, the various statistical operations can be used to mutate the abstracted parameter space in a reversible manner, so that the search space can be optimized and the search algorithm can be searched more efficiently (e.g., with respect to computational complexity) compared to the parameter space in which the various statistical operations have not been performed. Specifically, the application of these statistical operations can promote (when compared to the parameter space in which the various statistical operations have not been performed) one or more of the following: (a) dimensionality reduction, (b) conversion from a non-convex search space to a convex search space, or (c) increasing the connectivity and / or continuity of the scene space.

[0072] A parameter space internal representation (e.g., an internal parameter format) may be used in an apparatus operable for one or more of vehicle simulation, vehicle testing, or vehicle validation. The apparatus may include a memory and a processor operatively coupled to the memory. The processor may be configured to execute instructions to cause the apparatus to identify a first parameter set defined using a first scenario format. The first parameter set may include one or more of: (i) one or more first scenario format axes; (ii) one or more first scenario format relationships; or (iii) one or more first scenario format constraints. The processor may be configured to execute instructions to cause the apparatus to map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format contains one or more of: (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints. The processor may be configured to execute instructions to cause the apparatus to generate a test configuration object using the internal parameter set. The test configuration object may be configured to be linked to one or more objects. The processor may be configured to execute instructions to cause the apparatus to send the test configuration object to a test modality for testing.

[0073] A process flow 200 for vehicle simulation, testing or verification may be provided, such as Figure 2 . An input (e.g., a test specification 210) related to a scenario in a first scenario format (e.g., a scenario definition language) may be identified. The test specification 210 may include one or more of the following: a hardware-in-the-loop (HIL) test specification, a software-in-the-loop (SIL) test specification, or a field test specification. The HIL test specification may include, for example HIL Specifications. The SIL test specifications may be implemented using a first scenario format (eg, a scenario definition language), such as OSC 1.0, OSC 2.0, a different scenario language, or the like.

[0074] In one example, when the scenario is outside of a specific scenario language, the scenario can be imported or converted into the specific scenario language. When in the specific scenario language, the test specification 210 can be cross-compiled into an internal parameter format, which can be an intermediate still a markup language (YAML) format, as shown in operation 220, which can occur in a one-to-one operation. The internal parameter format can include one or more of the following: (i) one or more internal axes, (ii) one or more internal relations, (iii) one or more internal constraints, (iv) one or more internal metrics, or (v) one or more parsing operation information that facilitates conversion back to the first scenario format (e.g., the specific scenario language).

[0075] In another example, when the scene is in a scene-specific language, the internal parameter format can be configured to convert from the scene-specific language to the internal parameter format without importing or converting the scene into the scene-specific language. The internal parameter format can include additional conversion data that can be used to facilitate the conversion from the scene-specific language to the internal parameter format.

[0076] Thus, cross-compiling the test specification 210 into a first parameter set in an internal parameter format may cause one or more internal axes, one or more internal relations, or one or more internal constraints to be parsed into the internal parameter format, as shown in operation 230, which may occur in a one-to-one operation. The internal parameter format (e.g., a YAML format test specification) may be interchangeable between various test modalities (e.g., HIL, SIL, field test, or the like) and simulators (e.g., HIL simulators, SIL simulators, field test simulators, or the like). This internal parameter format may include one or more parsing operation information (e.g., relevant links and context information) that may be used to identify the compilation source in the internal parameter format to facilitate tracing the internal parameter set in the internal parameter format back to the first parameter set in the first parameter format (e.g., to identify information that may be specific to a specific scenario language). The internal parameter format may include one or more internal metrics for generating metrics using the axes when results are available after testing.

[0077] The first parameter set in the internal parameter format may be used to generate one or more of a test configuration object, a test run specification, or a test run result. The first parameter set in the internal parameter format, such as an intermediate YAML format, may be used to generate a test configuration object (i.e., a test configuration object), as shown in operation 240, which may be a one-to-many operation. The test configuration object may include a reference to one or more of a test specification, a parameter set in the internal parameter format, a test result, or the like to facilitate indexing data related to a particular test.

[0078] To initiate a test, a request including a first parameter set (and associated links) in an internal parameter format may be sent to a microservice to enqueue a test job, thereby generating a test run specification, as shown in operation 250, which may be a one-to-many operation. The test run specification may be a data packet including information for executing a test in a test environment. This information that may be used to execute a test in a test environment may include one or more of: (i) a test environment specification, (ii) a reference to one or more input files such as a test specification file, (iii) run-time parameters, or the like. A test configuration object may be formatted into a test modality format (e.g., a format specific to SIL, HIL, trace, or the like) for a test modality (e.g., SIL, HIL, trace, or the like). The formatted test configuration object may be sent to a test modality (e.g., SIL test, HIL test, trace test, or the like) associated with the test modality (e.g., SIL, HIL, trace, or the like).

[0079] Based on the requested test modality, a request to generate a test run specification may be forwarded to one or more interfaces of one or more test modalities, the one or more interfaces including: (i) an interface to a specific test scenario simulator, (ii) a plug-in interface to one or more of an external SIL simulator or an external HIL simulator, or (iii) a plug-in interface to an automated tracking test job queue. These one or more interfaces may perform other formatting from an internal parameter format (e.g., an intermediate scenario language) to a test specification. Other formatting may include one or more of the following: HIL device specifications, tracking test specifications, SIL specifications, or the like. These interfaces (e.g., plug-in interfaces) may submit requests to corresponding test modalities (e.g., SIL test modal for SIL test specifications, HIL test modal for HIL test specifications, tracking test modal for tracking test specifications, or the like) via an available web application protocol interface (API) (e.g., representational state transfer (REST), hypertext transfer protocol (HTTP), or the like).

[0080] The test run specification operation 250 , which may be a one-to-one operation, may generate a test run result as shown in operation 260 . The test run result may include one or more of the following: drive data, simulation data, or additional data that may be parsed based on the axis type.

[0081] Test modality format data may be converted to internal parameter format data at an internal parameter format converter plug-in. In one example, a converter plug-in may be used to receive external test run results (e.g., test run results for a specific scenario language). The external test run results may include various test modalities, including one or more of SIL, HIL, drive, or the like. The converter plug-in may receive various inputs, including: (i) internal parameter format, (ii) drive log or the like in any format. The converter plug-in may define a conversion from the drive log format to the internal parameter format. Metrics in the internal parameter format may be identified after testing.

[0082] The test run results may be used to generate test-independent data points using a one-to-many operation, as shown in operation 270. The test run results, which may include drive data, simulation data, or the like, may be in a time-based metric format. The time-based metric format may be collected from various test modalities, such as SIL, HIL, drive, tracking, or the like. The time-based metric format may be converted to an internal parameter format (e.g., an intermediate YAML format). The time-based metric format may be converted to an internal parameter format based on one or more of the following: (i) one or more internal axes, (ii) one or more internal relationships, or (iii) one or more internal constraints.

[0083] The internal parameter format converter plug-in can be configured to convert the axis data in the time-based metric signal format into the internal parameter format data, and write the axis data to the database. In one example, the time-based metric format can be converted into an intermediate YAML format using a YAML template for the axis to generate intermediate YAML format metrics. The post-processing microservice can calculate the relevant axis values ​​based on these intermediate YAML format metrics, which can be written to the database. Any axis that violates the constraints or relationships specified in the intermediate YAML format can be updated in the intermediate YAML format to include input axis values ​​that can be identified as untested.

[0084] As shown in operation 270, the test-independent data points that may be generated may be used to generate a data point probability distribution, which may be a many-to-many operation, as shown in operation 280. The data point probability distribution may be generated using one or more of: (i) an axis referenced in a particular data point probability distribution, (ii) a value referenced in a particular data point probability distribution, or (iii) a parameter referenced in a particular data point probability distribution.

[0085] The scenario definition language may include various scenario languages ​​(e.g., an open scenario format such as OSC or a dedicated scenario format such as a test specification for a HIL device). In one example, the scenario definition language may be an open scenario language. When using an open scenario language, a simulator (e.g., an autonomous vehicle (AV) simulator) may be configured to receive scenario format data and convert the scenario format data into internal parameter format data (e.g., intermediate YAML format data as shown in operation 220). The vehicle simulator may be configured to generate a test configuration object using the internal parameter format data (e.g., as shown in operation 240) and test the test configuration object using a test modality (e.g., as shown in operation 250). The vehicle simulator may be configured to further disclose with respect to generating the parameter space internal representation provided herein.

[0086] The vehicle simulator may be configured to generate various metrics (e.g., as shown in operation 260) to measure one or more of the coverage or performance of the vehicle simulator. The vehicle simulator may be configured to calculate parameter metrics (e.g., metrics calculated in a parameter format). The parameter metrics may be based on one or more of the parameter performance metrics or the parameter coverage metrics. The parameter metrics may be selected based on a specific use case type. The vehicle simulator may be configured to retrieve one or more axis values ​​for a test mode and calculate one or more axis-specific metrics based on the one or more axis values. The vehicle simulator may be configured to generate a test configuration object based on one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, and the one or more axis values ​​are selected based on the parameter metrics.

[0087] Parallel simulation using asynchronous messages

[0088] Functionality of vehicle simulation, testing, or verification may be configured using asynchronous information to generate simulations in parallel. Combining asynchronous information and parallelized simulation may enhance information gain in a specific time for a selected amount of computing resources. Parallelized simulation may facilitate large-scale information collection and generation of subsequent samples. Asynchronous information updates may facilitate collection from various test modalities (e.g., SIL, HIL, tracking) to determine subsequent tests. Combining parallel job scheduling with asynchronous information testing may enhance information gain when compared to a baseline approach because partial information may be used to update currently executing parallel tests without requiring random scheduling of subsequent test jobs and without waiting for completion information before adjusting subsequent test jobs.

[0089] Additionally, adding, adjusting, or terminating different test samples based on asynchronous information before completing testing of a particular test batch may reduce computational complexity to achieve convergence for a particular test case when compared to a baseline computational complexity in which no asynchronous information is used, no parallel testing is used, or neither asynchronous information nor parallel testing is used. The reduction in computational complexity to achieve convergence for a particular test case may occur due to increased testing of one or more of highly sensitive use cases, underdetermined use cases, edge cases, or failure-prone use cases when compared to the baseline computational complexity, and the reduction in computational complexity may occur due to reduced testing of less informative use cases that may provide less information when compared to the baseline test case. For example, the less informative use cases may include low sensitivity use cases, high coverage use cases, redundant test cases, use cases with sufficient test coverage, low failure rate use cases, or the like.

[0090] Operations on the sampling algorithm and parameter space may be updated asynchronously using parallelization assumptions. That is, when a scene is enqueued, an updated prior may be generated based on the collected data to enqueue subsequent scenes. A scene may contain data that has been collected asynchronously using an internal parameter format. The asynchronously collected data may contain one or more of simulated data or real-world data.

[0091] The vehicle simulator can be configured to Figure 3A One or more of the simulation, testing, or verification provided in the process flow 300a in . The vehicle simulator may be configured to test a first test sample batch in a test modality format. The first sample batch may include one or more first test samples that output one or more first test sample results. The vehicle simulator may be configured to identify one or more asynchronous test sample results of the plurality of first test samples before the first test sample batch is completed. The vehicle simulator may be configured to convert the one or more asynchronous test sample results from the test modality format to an internal parameter format. The vehicle simulator may be configured to adjust a second test sample batch based on the one or more asynchronous test sample results.

[0092] The vehicle simulator may be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, or the like) has been reached. For example, after testing the first and second batches, the coverage metric may not have reached a selected threshold. In this example, the vehicle simulator may be configured to adjust subsequent test sample batches (e.g., the third, fourth, etc.) until the coverage metric has reached the selected threshold. Testing of subsequent test sample batches may be repeated until the selected coverage metric has been reached.

[0093] The vehicle simulator can be configured to use one or more operations to process the test specifications for sending to the queue service 340a. The test specifications can include HIL test specifications, 1.0 or 2.0 test specification, a live test specification, or the like, as shown in block 302. The test specification may be provided to a scenario cross-compiler 310, which may compile the test specification into an internal parameter format (e.g., an intermediate YAML representation), as shown in block 312.

[0094] The internal parameter format (e.g., intermediate YAML representation) may be provided to one or more of the sampling service 320 or the linker service 330. The sampling service 320 may generate an object representation of one or more of: one or more axes (e.g., numeric or discrete axes), one or more relationships, or one or more constraints, as shown in block 322. The linker service 330 may receive the internal parameter format (e.g., YAML representation) and generate a test configuration object, as shown in block 332, which may provide links between one or more objects, results, or the like. The test configuration object, as shown in block 332, may be provided to the queue service 340a.

[0095] The queue service 340a may receive a test configuration object as shown in block 332 and user input (e.g., a user request) as shown in block 338. The queue service 340a may submit the test configuration object as shown in block 332 and the user input as shown in block 338 to one or more of the plug-ins or services for execution based on satisfaction of one or more constraints. The test configuration object as shown in block 342 may be provided to one or more of the HIL plug-in 344a, the SIL plug-in 344b, the field test plug-in 344c, or the test run specification 346.

[0096] The queue service 340a may receive a test trigger 348 from the smart job manager 340c. The smart job manager 340c may be configured to sample a data probability distribution (e.g., an augmented data probability distribution) for one or more test cases of a specific test modality (e.g., SIL, HIL, tracking, or the like). The one or more test cases may be one or more of the following: highly sensitive cases, underdetermined cases, edge cases, or failure-prone cases. A highly sensitive case may be a test case based on a variable that may affect the result to a greater extent when compared to a baseline test case. A baseline test case may be a test case that may be based on one or more of an average test sample result, a median test sample result, or the like. An underdetermined case may be a test case that does not contain enough test sample results to provide enough information about coverage, performance, or any other suitable metric. An edge case may be a test case that tests the boundary between one or more different test cases. A failure-prone case may be a test case that is more likely to cause a failure than a baseline test case.

[0097] The smart job manager 340c may be configured to identify previous test sample results, currently executed test samples (e.g., test jobs, simulations, or the like), and overall probability spaces for different test cases. The smart job manager 340c may receive test sample results that may change the probability distribution of data. The test sample results may include asynchronous test sample results. The smart job manager 340c may be configured to receive and / or identify asynchronous test sample results in a specified test modality format from a first test sample batch before the first test sample batch completes testing. The smart job manager 340c may be configured to convert the asynchronous test sample results to an internal parameter format when the asynchronous test sample results are in a test modality format (e.g., SIL, HIL, tracking, or the like).

[0098] The smart job manager 340c may be configured to adjust the second test sample batch based on one or more asynchronous test sample results. For example, the smart job manager 340c may be configured to cancel the currently executed test sample in the first test sample batch based on the asynchronous test sample result that can change the data probability distribution when a specific test sample result associated with the currently executed test sample is not used to change the data probability distribution, so as to release computing resources for other tests. The cancellation request may be sent from the smart job manager 340c to the parallel job scheduler 340b (e.g., Figure 3B 300b in the block diagram).

[0099] The smart job manager 340c may be configured to reduce the weight of the currently executed test sample when sampling the data probability distribution to maximize information gain and minimize redundant testing. The currently executed test sample may not have a test sample result included in the data probability distribution, but the currently executed test sample may have been queued before. Therefore, in order to avoid double counting of the currently executed test sample, the data probability distribution may be adjusted (e.g., its weight is reduced) based on the currently executed test sample.

[0100] The vehicle simulator may be configured to process the test specification using one or more operations to generate test run results 358, such as Figure 3B A test run specification 346 (eg, provided by one or more of the queue service 340a, the HIL plugin 344a, the SIL plugin 344b, or the field test plugin 344c) may be provided to the parallel job scheduler 340b.

[0101] The parallel job scheduler 340b can be configured to: (i) test one or more test sample batches in various test modality formats, and (ii) send the test sample batches to the various test modalities. The test sample batches can include one or more test samples that can be calculated to output one or more test sample results.

[0102] The parallel job scheduler 340b may be configured to identify various parameters related to job scheduling, including one or more of the following: randomness, learning rate decay, predicted sample size, test batch configuration parameters (e.g., hyperparameters related to past, present, and predicted future test executions). The parallel job scheduler may be implemented using various algorithms that may include hyperparameters that are based on the operating parameters of the algorithm.

[0103] The parallel job scheduler 340b may be configured to select a batch of test samples and provide the batch of test samples to one or more of the following under the test run specification 352: (i) one or more test modality plug-ins (e.g., HIL plug-in 354a, SIL plug-in 354b, field test plug-in 354c) for formatting into a test modality format (e.g., HIL, SIL, field test, or the like) to be input to a test modality (e.g., HIL simulation 356a, SIL simulation 356b, field test 356c), or (ii) or as input for a specific simulation 356d test. Results from one or more of the HIL simulation 356a, SIL simulation 356b, field test 356c, applied simulation 356d test, or the like may be collected as test run results 358. These test run results may be provided to the job post-processing service 450a (e.g., Figure 4A ).

[0104] The parallel job scheduler 340b may be configured to schedule one or more of the jobs, simulations, test batches, or the like to different test modalities, simulators, or the like by allocating software, networks, or other computing infrastructure in parallel to operate different jobs, simulations, test batches, or the like. The intelligent job manager 340c may be configured to send a request 359 to link and cancel a job to the parallel job scheduler 340b to release computing resources. The intelligent job manager 340c may be configured to generate a test specification (e.g., a newly generated test specification 311) that is processed into a test specification by being input to the scenario cross compiler 310 (e.g., Figure 3A ).

[0105] The parallel job scheduler 340b may be configured to receive a request 359 to join and cancel jobs to release computing resources. The request 359 to join and cancel jobs to release computing resources may be based on the asynchronous test sample results. Therefore, the parallel job scheduler 340b may adjust subsequent test sample batches based on the asynchronous test sample results. Alternatively or in addition, the parallel job scheduler 340b may be configured to adjust one or more subsequent test sample batches based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, or the like) has been reached.

[0106] The test run results 358 may include various metrics, axis values, or the like. The test run results 358 may include one or more of a stack or an environment log. The test run results 358 may include various data files, such as sensor and visualization data files. The test run results 358 may be provided to the post-processing service 350a. The post-processing service 350a may be configured to convert the axis data in the time-based metric signal format into the internal parameter format data at the internal parameter format converter plug-in. The axis data in the time-based metric signal format or the internal parameter format may be written to the database. When retrieving one or more axis values ​​for a test mode (e.g., SIL, HIL, tracking), and when calculating one or more axis-specific metric values ​​based on the one or more axis values, the database may be used. Data storage in the internal parameter format may facilitate efficient data retrieval when performing calculations. Alternatively or in addition, vectorized calculations may be performed to facilitate computational efficiency.

[0107] The test run results 358 may be processed in various operations to generate a data probability distribution to be sent to the smart job manager 340c. The data probability distribution may be calculated using parameter metrics (e.g., metrics based on one or more parameters). The parameter metrics may be calculated based on one or more of a parameter performance metric or a parameter coverage metric. The parameter metrics may be based on a specific use case type.

[0108] A parameter metric (e.g., a tunable exploration metric) can be used to explore the testable information space. In one example, the parameter metric can be tuned based on the difference between a random sample and samples that have already been collected. For example, for a uniform discrete distribution between 1 and 10, a heavy weighting of samples near the upper end of the range can indicate that samples near the lower end of the range can be explored. The tunable exploration metric can be adjusted as samples near the lower end of the distribution are collected. That is, the tunable exploration metric can be used to determine the extent of samples that fall outside the distribution, and can be adjusted as the distribution changes in response to sample collection.

[0109] Alternatively or additionally, the parallel job scheduler 340b may be configured to generate subsequent test sample batches based on one or more axis-specific metrics.The one or more axis-specific metrics may be based on one or more axis values ​​selected based on the parameter metrics.

[0110] The parallel job scheduler 340b may be configured to schedule jobs using asynchronous information (e.g., asynchronous test results). The parallel job scheduler 340b may be configured to schedule one or more different test samples in parallel and terminate, adjust, or add different test samples using asynchronous information (e.g., asynchronous test results). The parallel job scheduler 340b may terminate, adjust, or add different test samples using asynchronous information (e.g., asynchronous test results) without having a completion time for a particular test sample. The parallel job scheduler 340b may receive additional test samples for testing based on asynchronous information (e.g., asynchronous test results) before a previous test sample has completed testing.

[0111] An apparatus for testing, simulating, or validating a vehicle may include a memory and a processor. The processor may be configured to execute instructions to cause the apparatus to test a first test sample batch including one or more first test samples to output one or more first test sample results. The plurality of first test sample results may include completed test samples (which may be used as asynchronous information) and samples that have not yet been completed (and therefore may not be used as asynchronous information). The plurality of first test sample results may be generated using one or more different test modalities (e.g., SIL, HIL, tracking, or the like).

[0112] The processor may be configured to identify one or more asynchronous test sample results for the plurality of first test samples before the first test sample batch is completed. That is, the asynchronous test sample results may be completed test sample results even if the first test sample batch additionally includes unfinished test sample results. The asynchronous test sample results may be based on real-world data. The one or more asynchronous test sample results may be used to calculate a parameter metric.

[0113] The parameter metrics may include one or more of a parameter performance metric or a parameter coverage metric. The first sample batch may be terminated before completion based on the parameter metrics. The parameter metrics may be used to adjust the second test sample batch. The second test sample batch may be adjusted based on a test adjustment configuration parameter, the test adjustment configuration parameter including one or more of: a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, or the like.

[0114] The parallel job scheduler 340b may be configured to convert the plurality of first test samples from an internal parameter format to a test modality format. For example, the internal parameter format may be converted to a HIL format (e.g., using a HIL plug-in), to a SIL format (e.g., using a SIL plug-in), or to a field test format (e.g., using a field test plug-in).

[0115] When identifying the asynchronous information, the plurality of first test samples may be in a test modality format. The asynchronous information (e.g., one or more asynchronous test sample results) may be converted from the test modality format (when present in the test modality format) to an internal parameter format. Converting the asynchronous information to the internal parameter format may minimize computational complexity, computational time, and expense of adjusting a second test sample batch based on the asynchronous information.

[0116] measure

[0117] An apparatus for vehicle simulation, vehicle testing, or vehicle validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to execute instructions to cause the apparatus to identify a first set of parameters defined using a first scenario format. The first set of parameters may include one or more of: one or more first scenario format axes; one or more first scenario format relationships; or one or more first scenario format constraints.

[0118] like Figure 4A As shown in , functionality 400a is provided in which a post-processing service 450a may be configured to identify a first parameter set. The post-processing service 450a may be configured to map the first parameter set to an internal parameter set. The internal parameter set may include one or more of the following: one or more internal axes, one or more internal relationships, or one or more internal constraints. The post-processing service may be configured to transform the first parameter set (e.g., test run results) to generate an internal parameter set (e.g., test-independent data points 451). The internal parameter set (e.g., test-independent data points 451) may be input to a sampling service 450b.

[0119] The sampling service 450b may be configured to compute the data to be provided to the smart job manager 340c (e.g., Figure 3A). The sampling service 450b may be configured to identify one or more asynchronous test sample results for one or more first test samples before the first test sample batch has finished testing. The sampling service 450b may be configured to calculate parameter metrics. The parameter metrics may be based on one or more of a parameter performance metric or a parameter coverage metric. The sampling service 450b may be configured to select an algorithm set based on the parameter metrics. The sampling service 450b may be configured to apply an algorithm in the algorithm set to a second test sample batch before the first test sample batch has finished testing.

[0120] The sampling service 450b may be configured to generate and adjust an information space that may include an internal parameter set by adding data indexes based on one or more of the following: (i) one or more internal axis values, (ii) one or more internal relationship values, or (iii) one or more internal constraint values.

[0121] The sampling service 450b may be configured to select one or more of a set of axes or a subset of the set of axes based on a parameter metric. Results may be displayed on a display device based on one or more of the set of axes or a subset of the set of axes, where the set of axes is based on the parameter metric. One or more axis values ​​of the set of axes or a subset of the set of axes may be retrieved for a particular test modality (e.g., HIL, SIL, tracking, or the like), and one or more axis-specific metrics may be calculated based on the set of axes or a subset of the set of axes.

[0122] The sampling service 450b may be configured to calculate parameter metrics. The parameter metrics may be based on one or more of the parameter performance metrics or the parameter coverage metrics. The parameter metrics may be based on a specific use case type. The sampling service 450b may be configured to generate a test configuration object based on one or more axis specific metrics. The one or more axis specific metrics may be based on one or more axis values, which are selected based on the parameter metrics. The results may be displayed on a display device based on the parameter metrics.

[0123] like Figure 4B As shown in , various operations 400b show that the sampling service 450b can send the test-independent data points to the data index block 460a, which can be configured to process the test-independent data points 461 by building a primary index (as shown in block 462), and building a secondary index (i.e., view) (as shown in block 464). The primary index can include one or more of the following: (a) date, (b) axis, (c) test environment, (d) stack version, (e) other metadata, or the like (as shown in block 463).

[0124] The secondary index of build secondary index block 464 may be activated based on user input or based on the availability of computing resources. Build secondary index block 465 may generate a primary index group that may include filtered values ​​configured to select specific values, ranges, or the like of data points not relevant to the test.

[0125] The test-independent data points 461 may be sent to a characterization block 460b, which may process the test-independent data points 461 to generate results that are sent in a feedback loop to one or more of: (i) the test-independent data points 461, block 463, or block 465. Figure 4C , other operations 400c may include sending test-independent data points 461 to a characterization block 460b, which may be input to an apply kernel function block 466. A kernel function may be applied to the test-independent data points to generate a feature vector 468. Different techniques may be used to generate the feature vector 468 including one or more of: (i) linear, (ii) non-linear, (iii) deep learning-based functions, (iv) frequency-space transforms, (v) other transforms, or the like, as shown in block 467. The output of the characterization block 460b (e.g., feature vector 468) may be sent to a data indexing block 460a (e.g., to the test-independent data points 461, block 463, or block 465).

[0126] The sampling service 450b may use the data indexing and characterization 460b in block 460a to generate test-independent data points and indexes 453, such as Figure 4A . The test-independent data points and indexes 453 may be sent to an inference service 450c. The inference service 450c may be configured to filter the test-independent data points into a multidimensional distribution. The test-independent data points and indexes 453 may be filtered into a multidimensional distribution based on one or more of the correlations or based on various data point spaces that may use one or more primary indexes, secondary indexes, features, or the like to generate a data point probability distribution 480. The data point probability distribution 480 may be sent for distribution post-processing 460c.

[0127] A user interface may be provided to allow a user to view one or more of vehicle stack performance, test coverage, or uncertainty / test gaps at a glance via a web-based viewer. Via the user interface, a user may select any axis or combination of axes (e.g., performance metrics such as pass rate, time to collision, variables such as weather, driving speed, road type, etc.). Multidimensional charts of test results associated with a set of axes may be displayed via the user interface based on selections on the user interface (e.g., scatter matrices, cluster scatter plots, surface / contour plots, or the like). Via the user interface, filters may be applied and axes may be switched on these charts to iteratively filter the information space (e.g., stack performance, test coverage, uncertainty / test gaps, or the like).

[0128] In some instances, the user may click on suggestions for coverage gaps, highly correlated axes, relationships, clusters, and high uncertainty and low metric values. The user may select a combination of one or more of the axes, groupings, clusters, smoothing, interpolation, or statistical metrics to be viewed by sending a request to the sampling service 450b. The sampling service 450b may attempt to retrieve axis values ​​for a particular test modality (e.g., SIL, HIL, tracking, or the like). When there are no axes for a particular test modality, the axis values ​​may be calculated as requested, and the sampling service 450b may provide one or more of vehicle stack performance, test coverage, test gaps, or the like to the user interface. Alternatively or in addition, when there are axes for a particular test modality, the sampling service 450b may provide one or more of vehicle stack performance, test coverage, test gaps, or the like to the user interface.

[0129] When the axis values ​​have been calculated, the distribution post-processing 460c operation can be configured to facilitate various operations to send data to the probability distribution 470 operation and receive the data point probability distribution 480, such as Figure 4D The data point probability distribution 480 may be augmented with a kernel function, as shown in block 469a, or the data point probability distribution 480 may be used to compute a derived distribution, as shown in block 469b.

[0130] When data point probability distribution 480 is augmented with a kernel function, data point probability distribution 469c may be calculated. When data point probability distribution 480 is used to calculate a derived distribution, one or more of gradient 469d or density 469e may be calculated. One or more of data point probability distribution 469c, gradient 469d, or density 469e may be used to perform statistical analysis, as shown in block 469f.

[0131] The statistical analysis performed may include any suitable technique for generating a probability distribution, including one or more of: (i) clustering, (ii) regression, (iii) importance (e.g., based on a combination of coverage and / or performance), (iv) sensitivity analysis, or (v) entropy, as shown in block 469g. The distribution post-processing 460c operation may provide input to the probability distribution 470 operation for further processing by the smart job manager 440c.

[0132] The statistical analysis performed may include any suitable analysis technique for analyzing the probability distribution of data points. The statistical analysis performed may include grouping combinations of axis values ​​based on one or more of metadata tags, scenes, drive tags, locations, or the like for further calculations. Alternatively or in addition, derived metrics (e.g., confidence, confidence intervals, mean, minimum, maximum, counts, standard deviations, statistical entropy, or the like) may be calculated. Alternatively or in addition, clustering and correlation analysis may be performed to determine patterns. Example patterns may include linear kernel separators, kernel-based correlation analysis, K-means clustering, DB-CAN clustering, or the like. Alternatively or in addition, outliers, clusters, and correlation metrics may be linked to these results. Alternatively or in addition, smoothing and interpolation may be performed, including one or more of Gaussian smoothing, linear interpolation, nearest neighbor interpolation, cubic interpolation, quintic interpolation, or the like.

[0133] The user interface may provide page filtering for a particular scenario, simulation, driver, or the like. Input to the user interface may be submitted to the scenario cross compiler 310, such as Figure 3A , to compile into scenarios and test cases under a specific test modality (e.g., HIL, SIL, trace, or the like). The test modality specific plugin can be configured to compile the internal parameter set (in the internal parameter format) into a test modality format that can include the underlying external scenario.

[0134] like Figure 5 As shown in the block diagram 500 in FIG. 5 , the output from the distribution post-processing block 560 c may be sent to a probability distribution block 570 (e.g., annotated data probability distribution block 571). The annotated data probability distribution block 571 may be a data probability distribution that may be used to calculate a metric using, for example, an internal parameter set. The metric may include one or more of a performance metric or a coverage metric. The metric may be based on a usage example type. The metric may be calculated using one or more of: a confidence level, a confidence interval, a normalized format, or a sample density.

[0135] The annotated data probability distribution block 571 can be configured to identify specific driving situations based on one or more of: (i) importance (e.g., high sensitivity), (ii) density about importance (e.g., other data collection can be used), (iii) associated variables (e.g., regression), (iv) clusters (e.g., discrete patterns), or the like. These metrics (e.g., importance, density about importance, associated variables, clusters, or the like) can be used to select axes for additional testing.

[0136] The metric may be a unified metric that may be calculated based on a performance metric and a coverage metric. The unified metric may be based on one or more of: an objective function (e.g., a Boolean or continuous function) or real-world data. The unified metric may be a single metric that facilitates determination of performance and coverage over a unified parameter space. That is, the unified metric may be used for one or more of verification or auditing of one or more of performance or coverage relative to the unified parameter space. In one example, a query may include: "Can my stack achieve a 99.99% success rate in preventing a collision in an emergency braking event with a margin of error of 0.01%?" In response to the query, a set of statistical metrics (e.g., metrics, metrics, unified metrics, or the like) may be returned that accepts or rejects the query.

[0137] The quantification may be used to query and / or audit vehicle safety features.The annotated data probability distribution block 571 may be used to query vehicle safety features and calculate safety metrics based on the metrics in response to the vehicle safety feature query.

[0138] The metrics may be calculated using parallelization to maximize one or more of the operational design domain (ODD) coverage or ODD performance. The annotated data probability distribution block 571 may be used to determine the ODD coverage of the autonomous vehicle based on the metrics. The annotated data probability distribution block 571 may be used to determine the ODD performance of the autonomous vehicle based on the metrics. That is, the metrics may be used to evaluate the coverage and / or performance of a test set on a target ODD for a stack of vehicles (e.g., a stack of autonomous vehicles). The metrics may be used to select one or more sampling algorithms to optimize parallelization, thereby maximizing subsequent metrics (e.g., metrics, unified metrics, or the like).

[0139] The metrics may be used to determine specific tests that may be used to facilitate the collection of data to maximize one or more of performance or coverage of the unified parameter space. In one example, the metrics may indicate that light detection and ranging (LIDAR) may be used to increase one or more of performance or coverage of the unified parameter space. In another example, the user interface may accept input that may select a specific test based on a specific use case. A selected number of different algorithms and / or metrics associated with one or more tests may be provided to facilitate data collection. The user interface may receive input to select one or more of a specific test (e.g., LIDAR), a specific algorithm, a specific metric, or the like.

[0140] Metrics can be used with internal parameter formats to minimize data collection time and computing resources, and maximize the quality of the collected data. An apparatus for vehicle testing may include a memory and a processor operatively coupled to the memory. The processor may be configured to execute instructions to cause the apparatus to identify a first parameter set defined using a first scenario format, wherein the first parameter set includes one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints. The processor may be configured to execute instructions to cause the apparatus to map the first parameter set to an internal parameter set, wherein the internal parameter set includes one or more internal axes, one or more internal relationships, and one or more internal constraints. The processor may be configured to execute instructions to cause the apparatus to calculate metrics, wherein the metrics are based on one or more of performance metrics or coverage metrics. The apparatus may include a display device configured to display results based on the metrics. The metrics may be based on a usage paradigm type.

[0141] The annotated data probability distribution block 571 may be used to provide metrics to a sampling service (e.g., Figure 4A 450b) and / or inference services (e.g., Figure 4A ). The sampling service 450b and / or the inference service 450c may be configured to perform one or more of the following: (i) select an axis set based on a metric; (ii) display the results on a display device based on the axis set; (iii) select a subset of the axis set; or (iv) display a subset of the results on a display device based on the subset of the axis set. The sampling service 450b and / or the inference service 450c may be configured to: retrieve one or more axis values ​​for one or more test modalities, and calculate one or more axis-specific metrics based on the one or more axis values. The axis-specific metrics based on the one or more axis values ​​may be sent to the scenario cross-compiler 310 for sending to the linker service 330 to generate a test configuration object, wherein the one or more axis values ​​are based on, for example, a metric.

[0142] The metric may be used with the asynchronous test sample results to minimize test time and computing resources, and to maximize one or more of coverage or performance of subsequent test results. An apparatus for vehicle simulation, testing, or validation may include a memory and a processor operatively coupled to the memory. The processor may be configured to execute instructions to cause the apparatus to perform one or more of the following: (i) test a first test sample batch comprising one or more first test samples to output one or more first test sample results, (ii) identify one or more asynchronous test sample results for a plurality of first test samples before the first test sample batch is completed, (iii) calculate a metric based on the one or more asynchronous test sample results, wherein the metric comprises one or more of a performance metric or a coverage metric, or (iv) adjust a second test sample batch based on the metric.

[0143] The processor may be configured to determine one or more asynchronous test sample results based on parameter metrics, which may include one or more of parameter performance metrics or parameter coverage metrics. The asynchronous test sample results may be determined based on real-world data.

[0144] The first test sample results may be generated using one or more different test modalities (e.g., SIL, HIL, tracking, or the like). The first test samples may be converted from an internal parameter format to a test modality format. Asynchronous test sample results selectable from the first sample batch may be converted from the test modality format to the internal parameter format.

[0145] The metric can be used to terminate the first batch of samples before the first batch of samples has completed testing. The metric can be used to adjust the second test batch of samples based on test adjustment configuration parameters, wherein the test adjustment configuration parameters include one or more of the following: a selected sampling algorithm, a randomization parameter, or other hyperparameters, such as a learning rate decay parameter or a sample size parameter.

[0146] like Figure 5 As shown in FIG. 5 , an annotated data probability distribution block 571 may be sent to a test modality-specific annotated data probability distribution block 572, which may be configured to generate various test modality-specific distributions and sub-distributions, including: (i) a SIL-specific annotated distribution 573a, (ii) a HIL-specific annotated distribution 573b, (iii) a tracking-specific annotated distribution 573c, or the like. These test modality-specific annotated distributions (e.g., 573a, 573b, 573c) may be sent to an augmented test modality-specific data probability distribution block 574 to be sent to the smart job manager 540c.

[0147] The augmented test modality specific data probability distribution block 574 may include one or more of the distributions or sub-distributions of one or more of SIL, HIL, drive data, or the like. The augmented test modality specific data probability distribution block 574 may be augmented by various tests performed using a transformation function block 590 between two data probability distributions by including factors associated with the increased variance and / or uncertainty resulting from the transformation.

[0148] The annotated data probability distribution block 571 may be configured to send the annotated data probability distribution to the learning correlation block 580 for processing. The output from the learning correlation block 580 may be sent to a transformation function between two data probability distributions block 590, which may be configured to be output to an augmented test modality specific data probability distribution block 574.

[0149] Test the transfer function between modes

[0150] The apparatus for vehicle testing, simulation, and validation may include a memory and a processor. The processor may be configured to execute instructions to cause the apparatus to identify first test data based on a first test modality, wherein the test data includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data. For example, the learning correlation block 580 may be configured to receive data from the annotated data probability distribution block 571, such as Figure 5 As shown in Figure 6 As shown in , the data provided from the probability distribution block 670 to the learn correlation block 680 may be one or more augmented data point probability distributions (eg, augmented data point probability distribution 1 681a and augmented data point probability distribution 2 681b).

[0151] A transformation function may be generated to switch between different test data modalities. The processor may be configured to execute instructions to cause the device to calculate a transformation function, which is configured to transform the test data from a first test modality to a second test modality, wherein the second test modality is different from the first test modality. For example, the cross-distribution inference block 682 may be configured to generate transformation functions 686a, 686b: (i) a transformation function from an augmented data point probability distribution 1 681a to an augmented data point probability distribution 2 681b, and (ii) a transformation function from an augmented data point probability distribution 2 681b to an augmented data point probability distribution 1 681a. The transformation functions 686a, 686b may be generated using any suitable function including one or more of the following: (a) a non-parametric function 683 (e.g., a kernel), (b) a parametric function 684, or (c) a deep learning-based learning function 685.

[0152] The transformation function may be used to calculate test data under a test modality (e.g., second test data under a second test modality) that is different from the test modality of the received test data (e.g., first test data under a first test modality). The processor may be configured to calculate second test data based on the first test data using the transformation function, wherein the second test data includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data. The first test modality may be any suitable test modality including one or more of the following: SIL, HIL, tracking test, or the like. The second test modality may be any suitable test modality including one or more of the following: SIL, HIL, tracking test, or the like.

[0153] A plurality of metrics may be calculated to compare a difference between first test data and second test data and a transformation function between the first test data and the second test data, wherein the difference may occur due to generating the second test data using the first test data. The metrics may include: (i) an information loss metric to determine an amount of information loss that occurs when transforming the first test data into the second test data using the transformation function, (ii) a risk metric to determine an amount of risk that occurs when transforming the first test data into the second test data using the transformation function, or (iii) an uncertainty metric to determine an amount of uncertainty that occurs when transforming the first test data into the second test data using the transformation function.

[0154] Some of the metrics may be used to determine a reduction in one or more of test time or computational complexity that may occur when using a transformation function as compared to when not using the transformation function. The processor may be configured to calculate one or more of the transformed test time or the transformed computational complexity to calculate second test data using the transformation function based on the first test data. The processor may be configured to calculate one or more of the non-transformed test time or the non-transformed computational complexity to calculate the second test data without using the transformation function and the first test data. The processor may be configured to calculate one or more of the test time savings or the computational complexity savings based on the difference between: (i) the transformed test time and the non-transformed test time, or (ii) the transformed computational complexity and the non-transformed computational complexity.

[0155] Transformation functions may be used for classification to determine the ODD space. For example, when testing for different weather conditions, a sunny test scenario may be performed using a sunny day, and a rainy test scenario may be performed using a rainy day. The transformation function may be configured to transform one or more of the sunny test scenario or the rainy test scenario to collect data related to the test scenario for additional weather conditions (e.g., snowy, windy, dark, or the like).

[0156] The output of the skew correlation block 680 (e.g., the transformation function 690 between the two data probability distributions) may be provided as Figure 5 570 (e.g., annotated data probability distribution block 571) shown in FIG. 571 may be provided to an augmented test modality-specific data probability distribution 574. The output of the test modality-specific annotated data probability distribution (e.g., 573a, 573b, 573c) may be provided to the augmented test modality-specific data probability distribution 574. The augmented test modality-specific data probability distribution 574 may be sent to the smart job manager 540c.

[0157] Figure 7 A process flow of an example method 700 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 700 may be arranged according to at least one example described in the present disclosure.

[0158] The method 700 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 A processing device (e.g., processor) 1702 or another device, combination of devices, or system.

[0159] Method 700 may begin at block 705 , where processing logic may identify a first set of parameters defined using a first scene format, where the first set of parameters includes one or more first scene format axes, one or more first scene format relationships, and one or more first scene format constraints.

[0160] At block 710 , processing logic may map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format includes one or more internal axes, one or more internal relations, and one or more internal constraints.

[0161] At block 715 , processing logic may generate a test configuration object using the internal parameter set, wherein the test configuration object is configured to link to one or more objects.

[0162] At block 720 , processing logic may send the test configuration object to a test modality for testing.

[0163] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: identifying a compilation source in an internal parameter format to map an internal parameter set to a first parameter set; identifying a metric in an internal parameter format after a test; formatting a test configuration object into a test modality format for a test modality; sending the test configuration object to a test modality associated with the test modality; converting the test modality format data into the internal parameter format data at an internal parameter format converter plug-in; or converting the axis data in a time-based metric signal format into the internal parameter format data at an internal parameter format converter plug-in; or writing the axis data to a database. The test modality may include one or more of the following: a specific scenario language simulator, a software-in-the-loop (SIL) simulator plug-in interface, a hardware-in-the-loop (HIL) simulator plug-in interface, or a trace test plug-in interface. The first scenario format may include an open scenario format, which may include an OSC definition language.

[0164] Modifications, additions, or omissions may be made to method 700 without departing from the scope of the present disclosure. For example, in some instances, method 700 may include any number of other components that may not be explicitly shown or described.

[0165] Figure 8 A process flow of an example method 800 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 800 may be arranged according to at least one example described in the present disclosure.

[0166] The method 800 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0167] Method 800 may begin at block 805 , where processing logic may identify a first parameter set defined using a first scene format, where the first parameter set includes one or more first scene format axes, one or more first scene format relationships, and one or more first scene format constraints.

[0168] At block 810 , processing logic may map the first parameter set to an internal parameter set, where the internal parameter set includes one or more internal axes, one or more internal relations, and one or more internal constraints.

[0169] At block 815 , processing logic may calculate a parameter metric, wherein the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric.

[0170] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: select an axis set based on the parameter metric; display the results on the display device based on the axis set; or select a subset of the axis set; display a subset of the results on the display device based on the subset of the axis set; retrieve one or more axis values ​​for one or more test modalities; calculate one or more axis-specific metrics based on the one or more axis values; or generate a test configuration object based on the one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, and the one or more axis values ​​are selected based on the parameter metric. The parameter metric may be based on a usage example.

[0171] Modifications, additions, or omissions may be made to method 800 without departing from the scope of the present disclosure. For example, in some examples, method 800 may include any number of other components that may not be explicitly shown or described.

[0172] Fig. 9 A process flow of an example method 900 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 900 may be arranged according to at least one example described in the present disclosure.

[0173] The method 900 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0174] Method 900 may begin at block 905 , where processing logic may receive scene format data.

[0175] At block 910 , processing logic may convert scene format data into internal parameter format data.

[0176] At block 915 , processing logic may generate a test configuration object using the internal parameter format data.

[0177] At block 920 , processing logic may test the test configuration object using the test modality.

[0178] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: formatting a test configuration object into a test modality format for a test modality; sending the test configuration object to a test modality associated with the test modality, wherein the test modality includes one or more of a specific scenario format simulator, a software-in-the-loop (SIL) simulator plug-in interface, a hardware-in-the-loop (HIL) simulator plug-in interface, or a trace test plug-in interface; at an internal parameter format converter plug-in, converting test modality format data into internal parameter format data; at the internal parameter format converter plug-in, converting axis data in a time-based metric signal format into internal parameter format data; writing the axis data to a database; calculating parameter metrics, wherein the parameter metrics are based on one or more of parameter performance metrics or parameter coverage metrics, wherein the parameter metrics are based on a usage example; retrieving one or more axis values ​​for the test modality; calculating one or more axis-specific metrics based on the one or more axis values; generating a test configuration object based on the one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on the one or more axis values, and the one or more axis values ​​are selected based on the parameter metrics.

[0179] Modifications, additions, or omissions may be made to method 900 without departing from the scope of the present disclosure. For example, in some instances, method 900 may include any number of other components that may not be explicitly shown or described.

[0180] Fig.10 A process flow of an example method 1000 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1000 may be arranged according to at least one example described in the present disclosure.

[0181] The method 1000 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0182] Method 1000 may begin at block 1005 , where processing logic may test a first test sample batch including one or more first test samples to output one or more first test sample results.

[0183] At block 1010 , processing logic may identify one or more asynchronous test sample results for a plurality of first test samples prior to completion of a first test sample batch.

[0184] At block 1015 , processing logic may calculate a parameter metric based on the one or more asynchronous test sample results, wherein the parameter metric includes one or more of a parameter performance metric or a parameter coverage metric.

[0185] At block 1020, processing logic may adjust the second test sample batch based on the parameter metric.

[0186] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: determine one or more asynchronous test sample results based on a parameter metric, the parameter metric including one or more of a parameter performance metric or a parameter coverage metric; determine the asynchronous test sample results based on real-world data; terminate the first sample batch based on the parameter metric; convert a plurality of first test samples from an internal parameter format to a test modality format; convert one or more asynchronous test sample results from the test modality format to an internal parameter format; adjust the second test sample batch based on a test adjustment configuration parameter, the test adjustment configuration parameter including one or more of the following: a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, or the like. The plurality of first test sample results may be generated using one or more different test modalities. Alternatively or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, or the like) has been reached.

[0187] Modifications, additions, or omissions may be made to method 1000 without departing from the scope of the present disclosure. For example, in some instances, method 1000 may include any number of other components that may not be explicitly shown or described.

[0188] Fig.11 A process flow of an example method 1100 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1100 may be arranged according to at least one example described in the present disclosure.

[0189] The method 1100 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0190] The method 1100 may begin at block 1105 where processing logic may identify one or more asynchronous test sample results for one or more first test samples before a first test sample batch has completed testing.

[0191] At block 1110 , processing logic may calculate a parameter metric, where the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric.

[0192] At block 1115 , processing logic may select a set of algorithms based on the parameter metrics.

[0193] At block 1120 , processing logic may apply an algorithm from the set of algorithms to a second batch of test samples before the first batch of test samples has completed testing.

[0194] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: calculate a parameter metric including one or more of the following: a confidence level, a confidence interval, a normalized metric, or a sample density; select an axis set based on the parameter metric; display the results on a display device based on the axis set; select a subset of the axis set; display a subset of the results on a display device based on the subset of the axis set; retrieve one or more axis values ​​for one or more test modalities; calculate one or more axis-specific metrics based on one or more axis values; or generate a test configuration object based on one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, and the one or more axis values ​​are selected based on the parameter metric. The parameter metric may be based on a usage example. The parameter metric may be calculated using one or more of an objective function or real-world data. Alternatively or in addition, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (e.g., a convergence metric, a confidence metric, a coverage metric, a sampling metric, or the like) has been reached.

[0195] Modifications, additions, or omissions may be made to method 1100 without departing from the scope of the present disclosure. For example, in some examples, method 1100 may include any number of other components that may not be explicitly shown or described.

[0196] Fig.12 A process flow of an example method 1200 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1200 may be arranged according to at least one example described in the present disclosure.

[0197] The method 1200 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0198] The method 1200 may begin at block 1205 where processing logic may test a first test sample batch in a test modality format, where the first sample batch includes one or more first test samples to output one or more first test sample results.

[0199] At block 1210 , processing logic may identify one or more asynchronous test sample results for a plurality of first test samples prior to completion of a first test sample batch.

[0200] At block 1215 , processing logic may convert one or more asynchronous test sample results from a test modality format to an internal parameter format.

[0201] At block 1220 , processing logic may adjust the second test sample batch based on the one or more asynchronous test sample results.

[0202] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: formatting a second test sample batch into a test modality format for a test modality; sending the second test sample batch to a test modality associated with the test modality, wherein the test modality includes one or more of a specific scenario format simulator, a software-in-the-loop (SIL) simulator plug-in interface, a hardware-in-the-loop (HIL) simulator plug-in interface, or a trace test plug-in interface; at an internal parameter format converter plug-in, converting axis data in a time-based metric signal format into internal parameter format data; writing the axis data to a database; calculating parameter metrics, wherein the parameter metrics are based on one or more of parameter performance metrics or parameter coverage metrics, wherein the parameter metrics are based on a usage example; retrieving one or more axis values ​​for the test modality; or calculating one or more axis-specific metrics based on one or more axis values; generating a second test sample batch based on the one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, and the one or more axis values ​​are selected based on the parameter metrics. Alternatively or additionally, one or more subsequent test sample batches may be adjusted based on one or more previous test sample batches until a metric (eg, a convergence metric, a confidence metric, a coverage metric, a sampling metric, or the like) has been reached.

[0203] Modifications, additions, or omissions may be made to method 1200 without departing from the scope of the present disclosure. For example, in some instances, method 1200 may include any number of other components that may not be explicitly shown or described.

[0204] Fig.13 A process flow of an example method 1300 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1300 may be arranged according to at least one example described in the present disclosure.

[0205] The method 1300 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17The processing device 1702 or another device, combination of devices, or system.

[0206] Method 1300 may begin at block 1305 , where processing logic may calculate an internal parameter set including one or more internal axes, one or more internal relations, and one or more internal constraints.

[0207] At block 1310 , processing logic may calculate a metric using the internal parameter set, where the metric includes one or more of a performance metric or a coverage metric.

[0208] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: calculating a metric based on a usage paradigm; calculating a metric using one or more of the following: a confidence level, a confidence interval, a normalized format, or a sample density; determining an operational design domain (ODD) coverage for a vehicle based on the metric; determining an operational design domain (ODD) performance for a vehicle based on the metric; querying a vehicle safety feature; calculating a safety metric based on the metric in response to a vehicle safety feature query; calculating a unified metric based on a performance metric and a coverage metric; calculating a unified metric based on one or more of the following: an objective function or real-world data; or calculating the metric using parallelization to maximize one or more of operational design domain (ODD) coverage or ODD performance.

[0209] Modifications, additions, or omissions may be made to method 1300 without departing from the scope of the present disclosure. For example, in some examples, method 1300 may include any number of other components that may not be explicitly shown or described.

[0210] Fig.14 A process flow of an example method 1400 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1400 may be arranged according to at least one example described in the present disclosure.

[0211] The method 1400 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0212] Method 1400 may begin at block 1405 , where processing logic may identify a first parameter set defined using a first scene format, where the first parameter set includes one or more first scene format axes, one or more first scene format relationships, and one or more first scene format constraints.

[0213] At block 1410 , processing logic may map the first parameter set to an internal parameter set, where the internal parameter set includes one or more internal axes, one or more internal relations, and one or more internal constraints.

[0214] At block 1415 , processing logic may calculate a metric, where the metric is based on one or more of a performance metric or a coverage metric.

[0215] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: select an axis set based on the metric; display the results on a display device based on the axis set; or select a subset of the axis set; display a subset of the results on a display device based on the subset of the axis set; retrieve one or more axis values ​​for one or more test modalities; calculate one or more axis-specific metrics based on the one or more axis values; or generate a test configuration object based on the one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, the one or more axis values ​​being selected based on the metric. The metric may be based on a usage paradigm.

[0216] Modifications, additions, or omissions may be made to method 1400 without departing from the scope of the present disclosure. For example, in some examples, method 1400 may include any number of other components that may not be explicitly shown or described.

[0217] Fig.15 A process flow of an example method 1500 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1500 may be arranged according to at least one example described in the present disclosure.

[0218] The method 1500 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0219] Method 1500 may begin at block 1505 , where processing logic may test a first test sample batch including one or more first test samples to output one or more first test sample results.

[0220] At block 1510 , processing logic may identify one or more asynchronous test sample results for a plurality of first test samples prior to completion of a first test sample batch.

[0221] At block 1515 , processing logic may calculate a metric based on the one or more asynchronous test sample results, where the metric includes one or more of a performance metric or a coverage metric.

[0222] At block 1520 , processing logic may adjust the second test sample batch based on the metric.

[0223] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: determine one or more asynchronous test sample results based on parameter metrics, the parameter metrics including one or more of parameter performance metrics or parameter coverage metrics; determine one or more asynchronous test sample results based on real-world data; terminate the first sample batch based on the metrics; convert the plurality of first test samples from an internal parameter format to a test modality format; convert one or more asynchronous test sample results from the test modality format to an internal parameter format; or adjust the second test sample batch based on a test adjustment configuration parameter, the test adjustment configuration parameter including one or more of the following: a learning rate decay parameter, a sample size parameter, a hyperparameter, an algorithm selection parameter, or the like. The first test sample results may be generated using one or more different test modalities.

[0224] Modifications, additions, or omissions may be made to method 1500 without departing from the scope of the present disclosure. For example, in some instances, method 1500 may include any number of other components that may not be explicitly shown or described.

[0225] Fig.16 A process flow of an example method 1600 that may be used for vehicle testing, simulation, or validation according to at least one example described in the present disclosure is shown. The method 1600 may be arranged according to at least one example described in the present disclosure.

[0226] The method 1600 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a computer system or a dedicated machine), or a combination of both. Fig.17 The processing device 1702 or another device, combination of devices, or system.

[0227] Method 1600 may begin at block 1605 where processing logic may identify first test data based on a first test modality, where the first test data includes one or more of first coverage test data, first performance test data, first metric distribution test data, or first uncertainty test data.

[0228] At block 1610 , processing logic may calculate a transformation function configured to transform first test data from a first test modality to a second test modality, where the second test modality is different from the first test modality.

[0229] At block 1615 , processing logic may calculate second test data based on the first test data using a transformation function, wherein the second test data includes one or more of second coverage test data, second performance test data, second metric distribution test data, or second uncertainty test data.

[0230] The processor may be further configured to execute instructions to cause the device to perform one or more of the following: calculate an information loss metric based on one or more of the first test data or the second test data, wherein the information loss metric is the amount of information loss of the second test data compared to the first test data; calculate a risk metric based on one or more of the first test data or the second test data, wherein the risk metric is the amount of risk of the second test data compared to the risk of the first test data; calculate an uncertainty metric based on one or more of the first test data or the second test data, wherein the uncertainty metric is the amount of uncertainty of the second test data compared to the uncertainty of the first test data; calculate a transformed test time based on the first test data using a transformation function to calculate the second test data; calculate a non-transformed test time without using the transformation function and the first test data to calculate the second test data; or calculate a test time saving based on the difference between the transformed test time and the non-transformed test time. The first test modality may be one or more of the following: software in the loop (SIL), hardware in the loop (HIL), tracking test. The second test modality may be one or more of the following: software in the loop (SIL), hardware in the loop (HIL), tracking test.

[0231] Modifications, additions, or omissions may be made to method 1600 without departing from the scope of the present disclosure. For example, in some examples, method 1600 may include any number of other components that may not be explicitly shown or described.

[0232] For the sake of simplicity of explanation, the methods and / or process flows described herein are depicted and described as a series of actions. However, actions according to the present disclosure can occur in various orders and / or simultaneously, and have other actions not presented and described herein. In addition, not all of the described actions can be used to implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method can be represented as a series of interrelated states via a state diagram or an event instead. In addition, the methods disclosed in this specification can be stored on articles such as non-transitory computer-readable media to facilitate the transmission and transfer of such methods to a computing device. As used herein, the term article is intended to cover a computer program that can be accessed from any computer-readable device or storage medium. Although shown as discrete boxes, various boxes can be divided into additional boxes, combined into fewer boxes, or the boxes can be removed depending on the desired implementation.

[0233] In some examples, a vehicle (e.g., AV) simulation, testing, and validation system may include an environment representation creation module, which may include code and routines configured to enable a computing device to perform one or more operations of generating a 3D environment representation, preparing and generating a scene, simulating a scene, and validating and / or evaluating a simulation of a scene. Additionally or alternatively, the environment representation creation module may be implemented using hardware, including a processor, a microprocessor (e.g., to perform or control the performance of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, the environment representation creation module may be implemented using a combination of hardware and software. In the present disclosure, operations described as being performed by the environment representation creation module may include operations that the environment representation creation module may direct the corresponding system to perform.

[0234] In some examples, the environment representation creation module may be configured to generate a 3D environment representation. The environment representation creation module may generate the 3D environment representation according to any suitable 3D modeling technology. In some examples, the environment representation creation module may use map data as input data to generate the 3D environment representation. For example, the 3D environment of the 3D environment representation may represent a geographic area represented by the map data.

[0235] In some examples, the 3D environment representation may include a 3D model of one or more objects in a geographic area as described by the map data. For example, the 3D environment representation may include a complete 3D model of a simulated driving environment.

[0236] A vehicle (e.g., AV) simulation, test, and validation system may include machine learning circuitry and / or software to more efficiently and proactively identify failure cases. A vehicle (e.g., AV) simulation, test, and validation system may include circuitry and / or software for tailoring, including parameter-based tailoring. A vehicle (e.g., AV) simulation, test, and validation system may include circuitry and / or software to enforce runtime constraints on a scenario, simulation, or within a 3D environment representation.

[0237] Fig.17A graphical representation of a machine in the example form of a computing device 1700 is shown within which a set of instructions may be executed for causing the machine to perform any one or more of the methods discussed herein. The computing device 1700 may include a rack server, a router computer, a server computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, or any computing device having at least one processor, etc., within which a set of instructions may be executed for causing the machine to perform any one or more of the methods discussed herein. In an alternative example, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine may operate as a server machine in a client-server network environment. In addition, although only a single machine is shown, the term "machine" may also include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0238] The example computing device 1700 includes a processing device (e.g., a processor) 1702, a main memory 1704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM)), a static memory 1706 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 1716 that communicate with each other via a bus 1708.

[0239] Processing device 1702 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, processing device 1702 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. Processing device 1702 may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. Processing device 1702 is configured to execute instructions 1726 for performing the operations and steps discussed herein.

[0240] The computing device 1700 may further include a network interface device 1722 that may communicate with the network 1718. The computing device 1700 may also include a display device 1710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1712 (e.g., a keyboard), a cursor control device 1714 (e.g., a mouse), and a signal generating device 1720 (e.g., a speaker). In at least one example, the display device 1710, the alphanumeric input device 1712, and the cursor control device 1714 may be combined into a single component or device (e.g., an LCD touch screen).

[0241] The data storage device 1716 may include a computer-readable storage medium 1724 on which is stored one or more sets of instructions 1726 embodying any one or more of the methodologies or functions described herein. The instructions 1726 may also reside, completely or at least partially, within the main memory 1704 and / or within the processing device 1702 during execution thereof by the computing device 1700, the main memory 1704 and the processing device 1702 also constituting computer-readable media. The instructions may further be transmitted or received over the network 1718 via the network interface device 1722.

[0242] Although computer-readable storage medium 1724 is shown as a single medium in the example, the term "computer-readable storage medium" may include a single medium or multiple media (such as a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" may also include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure. The term "computer-readable storage medium" may therefore be considered to include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0243] Examples

[0244] Examples of the present disclosure are provided below.

[0245] Example 1: Lane support using an open scene

[0246] In this example, the target vehicle and the host vehicle are described relative to a route. The host vehicle is driven along the route and is initially in the same lane as the target vehicle. The target vehicle accelerates towards the host vehicle and has a distance of about 0.1 meters to the lane marking on the right. At the beginning of the scenario, the host vehicle has a distance of about 100 meters between itself and the target vehicle and a distance of about 0.5 meters to the lane marking on the right. During the scenario, the host vehicle switches to a different lane. At the end of the scenario, the host vehicle switches back to the same lane as the target vehicle.

[0247] An example of the OpenSCENARIO 2.0 scene format is provided in Table I.

[0248] Table I: Behind Lane Support

[0249]

[0250] Example 2: OpenSCENARIO to internal parameter format conversion

[0251] The scene description language OpenSCENARIO can be converted into an internal parameter format.

[0252] The OpenSCENARIO code provided in Table I includes: (i) axes (e.g., vehicle, route, lane, lateral direction, position, speed, and the like), (ii) relationships (e.g., target vehicle and host vehicle in the same lane), and (iii) constraints (e.g., at least 2 lanes). The OpenSCENARIO code can be mapped to an internal parameter format by identifying the OpenSCENARIO axes, relationships, and constraints and converting those axes, relationships, and constraints into an internal parameter format (e.g., an intermediate YAML format).

[0253] Example 3: Testing in parallel using asynchronous messages

[0254] Two different scenarios in the OpenSCENARIO language can be converted into an internal parameter format. The two scenarios can be tested in the same batch as the first test and the second test. The first test based on the first scenario can complete the test before the second test based on the scenario completes the test. The results of the first test based on the first scenario can be used in the third test based on the second scenario.

[0255] The first scenario is provided in Table I, and the second scenario is provided in Table II.

[0256] Table II: Staying Ahead Lane Support

[0257]

[0258] In the second scenario, the host vehicle does not change lanes in response to the target vehicle accelerating behind it in the same lane. The host vehicle accelerates from 100 kph to 130 kph. The target vehicle starts the scenario at 150 kph. This scenario can be used to test the response of an autonomous vehicle to an approaching car.

[0259] The first test results from the first scenario may indicate that the host vehicle did not change lanes and allowed the host vehicle to crash into the target vehicle from behind. These first test results, when completed before the second test results are completed, may be used to adjust the third test based on the second scenario. Knowing that the host vehicle was unable to change lanes and the target car crashed into the host vehicle may be used to modify the third scenario by, for example, reducing the speed of the target vehicle to test whether the host vehicle will be able to sense the target vehicle and avoid a collision.

[0260] Example 4: Parameter metrics using internal parameter format and asynchronous information

[0261] The scenarios set forth in Tables I ("falling behind") and II ("staying ahead") may be used to compute parameter metrics. Converting the scenarios to an internal parameter format (e.g., an intermediate YAML format) may enhance the performance and information gain received for parameter metrics (e.g., one or more of a parameter performance metric or a parameter coverage metric).

[0262] Scenarios represented using OpenSCENARIO can be converted to an internal parameter format to facilitate conversion of scenarios to different test modalities (e.g., SIL, HIL, tracking), and test results from different test modalities can be converted back to the internal parameter format. Metrics from different test modalities can be compared using the same language. Therefore, parameter coverage metrics and parameter performance metrics can be calculated with greater accuracy and precision than calculations based on different languages.

[0263] For the scenarios described in Falling Behind and Staying Ahead, one performance metric may be the frequency with which the host vehicle crashes due to: (i) changing lanes to allow the target vehicle to pass, or (ii) staying in the same lane and increasing speed. A coverage metric may be calculated based on the number of different scenarios (e.g., different weather, car speeds, road types, or the like) related to these scenarios in Falling Behind and Staying Ahead that are tested based on test data from the scenarios in Falling Behind and Staying Ahead.

[0264] Converting the OpenSCENARIO format to an internal parameter format (e.g., an intermediate YAML format) can help combine data from the OpenSCENARIO format with data from different scenario languages ​​(i.e., non-OpenSCENARIO languages) and calculate metrics based on the combined data. Different scenario languages ​​can collect data related to weather conditions and road types for the host vehicle and the target vehicle in different scenarios (e.g., a combined lane change scenario, in which the host vehicle and the target vehicle change lanes at approximately the same time while traveling within a selected distance of each other). The data from this scenario can be converted to an internal parameter format (e.g., an intermediate YAML format) and combined with data from the scenarios described in Falling Behind and Staying Ahead. When the scenarios described in Tables I and II are combined with the combined lane change before calculating the coverage metric, the sum of the coverage metric provided by Falling Behind and Staying Ahead and the coverage metric provided by the combined lane change scenario may not be the same as the coverage metric.

[0265] The asynchronous test results can be used to increase information gain and reduce computational complexity and test time. For situations where the fall behind scenario and the combined lane change scenario complete testing before the stay ahead scenario completes testing, the fall behind and combined lane change scenarios can be converted to an internal parameter format (e.g., an intermediate YAML format) before computing performance and / or coverage metrics. The resulting performance and / or coverage metrics can be used to select a specific algorithm to apply to a subsequent test batch before the test data from the stay ahead scenario has completed.

[0266] Example 5: Metrics using internal parameter format and asynchronous messages

[0267] Falling behind, staying ahead, and combined lane change scenarios may be used with metrics. Metrics may be calculated using one or more of confidence levels, confidence intervals, normalized formats, or sample density. For example, a falling behind scenario may have a 2% collision rate, a staying ahead scenario may have a 3% collision rate, and a combined lane change scenario may have a 5% collision rate. Confidence levels and intervals may be calculated for these scenarios. A falling behind scenario with a 2% collision rate may have a confidence interval of 2.5% to 3.5%, and a 95% confidence level. A staying ahead scenario with a 3% collision rate may have a confidence interval of 3.5% to 4.5%, and a 97% confidence level. A combined lane change scenario with a 5% collision rate may have a confidence interval of 3% to 8%, and a 85% confidence level.

[0268] The normalized format may be calculated by combining three different scenarios using the internal parameter format and normalizing the data so that the area under the curve is approximately equal to 1. The sample density may be calculated based on the sample density of each of the scenarios.

[0269] The metrics may be combined into a single metric (e.g., a unified metric) that combines the coverage metric with the performance metric. The single metric may be used to determine what to test. For example, a coverage metric based on the fall behind, stay ahead, and combined lane change scenarios may indicate that coverage is not provided for a scenario where two vehicles traveling at 300 kph during a snowstorm simultaneously change lanes into the same lane, but the single metric may calculate that the scenario does not need to be tested because the performance (e.g., collision) is known.

[0270] Example 6: Transformation Function

[0271] First test data calculated using a first test modality may be transformed into second test data for a second test modality. For example, the first test data may be SIL test data and the second test data may be HIL data. A transformation function may be determined between the first test modality and the second test modality using a suitable machine learning technique, such as supervised machine learning (e.g., regression) using training data for the first and second test modalities. An objective function may be optimized to generate one or more functions for transforming between the test modalities.

[0272] For the trailing, leading, and combined lane change scenarios, the test modality may include SIL test data. The SIL test data may be converted to HIL test data using a transformation function. Using a transformation function to convert SIL test data to HIL test data may help reduce test time.

[0273] In some instances, the various components, modules, engines, and services described herein may be implemented as objects or processes executed on a computing system (e.g., as separate threads). Although some of the systems and methods described herein are generally described as being implemented in software (stored on hardware and / or executed by hardware), specific hardware implementations or combinations of software and specific hardware implementations are possible and contemplated.

[0274] The terms used herein, and especially in the following claims (e.g., the bodies of the following claims), are generally intended to be “open” terms (e.g., the term “including” should be interpreted as “including, but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “including, but not limited to,” etc.).

[0275] Additionally, if a specific number of introduced claim recitations is intended, such intent will be expressly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted as implying that the introduction of a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to instances containing only one such recitation, even when the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"); the same applies to situations where definite articles are used to introduce claim recitations.

[0276] In addition, even if a specific number of claim recitations introduced is explicitly recited, it should be understood that such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations" without other modifiers means at least two recitations or two or more recitations). Moreover, in those cases where a convention similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, generally speaking, such constructions are intended to include only A, only B, only C, A and B, A and C, B and C, or A, B, and C, etc. For example, use of the term "and / or" is intended to be interpreted in this manner.

[0277] In addition, any transitional word or phrase presenting two or more alternative terms, whether in the specification, claims or drawings, should be understood to contemplate the possibility of including one, either or both of the terms. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0278] In addition, the use of the terms "first", "second", "third", etc. in this document is not necessarily used to imply a specific order or number of elements. In general, the terms "first", "second", "third", etc. are used as general identifiers to distinguish different elements. In the absence of an indication that the terms "first", "second", "third", etc. imply a specific order, these terms should not be understood to imply a specific order. In addition, in the absence of an indication that the terms "first", "second", "third", etc. imply a specific number of elements, these terms should not be understood to imply a specific number of elements. For example, a first widget may be described as having a first side, and a second widget may be described as having a second side. Use of the term "second side" with respect to the second widget may be used to distinguish such side of the second widget from the "first side" of the first widget, without implying that the second widget has two sides.

[0279] In addition, all examples and conditional language described herein are intended to help readers understand the teaching objectives of the present invention and the concepts contributed by the inventors to the development of the art, and should be interpreted as not being limited to such specific examples and conditions. Although examples of the present disclosure have been described in detail, it should be understood that various changes, substitutions and modifications may be made to the present disclosure without departing from the spirit and scope of the present disclosure.

Claims

1. A device for vehicle simulation, comprising: Memory; as well as a processor operatively coupled to the memory, the processor configured to execute instructions to cause the apparatus to: A first parameter set defined using a first scene format is identified, wherein the first parameter set includes: one or more first scene format axes, one or more first scene format relations, and one or more first scene format constraints; Mapping the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format comprises: One or more internal axes, one or more inner relations, and one or more internal constraints; generating a test configuration object using the internal parameter set, wherein the test configuration object is configured to be linked to a plurality of objects; and Send the test configuration object to the test modality for use in testing.

2. The apparatus of claim 1 , wherein the processor is further configured to execute instructions to cause the apparatus to: Compilation source in the intrinsic parameter format is identified to map the intrinsic parameter set to the first parameter set.

3. The apparatus of claim 1 , wherein the processor is further configured to execute instructions to cause the apparatus to: Metrics in the internal parameter format are identified after testing.

4. The apparatus of claim 1, wherein the test modality comprises one or more of: a software-in-the-loop (SIL) simulator plug-in interface, a hardware-in-the-loop (HIL) simulator plug-in interface, or a trace test plug-in interface.

5. The apparatus of claim 1, wherein the processor is further configured to execute instructions to cause the apparatus to: formatting the test configuration object into a test modality format for the test modality; and Send the test configuration object to the test modality.

6. The apparatus of claim 1, wherein the first scene format comprises one or more of: an Open Scene (OSC) definition language.

7. The apparatus of claim 1, wherein the processor is further configured to execute instructions to cause the apparatus to: At the internal parameter format converter plug-in, the test modality format data is converted into the internal parameter format data.

8. The apparatus of claim 1, wherein the processor is further configured to execute instructions to cause the apparatus to: At the internal parameter format converter plug-in, converting the axis data in the time-based metric signal format into internal parameter format data; and The axis data is written to a database.

9. A device for vehicle testing, comprising: Memory; a processor operatively coupled to the memory, the processor configured to execute instructions to cause the apparatus to: A first parameter set defined using a first scene format is identified, wherein the first parameter set includes: one or more first scene format axes, one or more first scene format relations, and one or more first scene format constraints; Mapping the first parameter set to an internal parameter set, wherein the internal parameter set comprises: One or more internal axes, one or more inner relations, and One or more internal constraints; and A parameter metric is calculated, wherein the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric.

10. The apparatus of claim 9, wherein the parameter metric is based on a usage paradigm.

11. The apparatus of claim 9, wherein the processor is further configured to execute instructions to cause the apparatus to: selecting a set of axes based on the parameter metric; or displaying the result on a display device based on the set of axes; or Select a subset of the set of axes; or A subset of the results is displayed on the display device based on the subset of the set of axes.

12. The apparatus of claim 9, wherein the processor is further configured to execute instructions to cause the apparatus to: retrieving one or more axis values ​​for one or more test modalities; and One or more axis-specific metrics are calculated based on the one or more axis values.

13. The apparatus of claim 9, wherein the processor is further configured to execute instructions to cause the apparatus to: A test configuration object is generated based on one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values ​​selected based on the parameter metric.

14. A non-transitory computer-readable storage medium comprising computer-executable instructions that, when executed by one or more processors, cause a vehicle simulator to: Receive scene format data; Converting the scene format data into internal parameter format data; Generate a test configuration object using the internal parameter format data; and Test the test configuration object using the test mode.

15. The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: formatting the test configuration object into a test modality format for the test modality; and Sending the test configuration object to the test modality, The test mode includes one or more of a software-in-the-loop (SIL) simulator plug-in interface, a hardware-in-the-loop (HIL) simulator plug-in interface or a trace test plug-in interface.

16. The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: At the internal parameter format converter plug-in, the test modality format data is converted into the internal parameter format data.

17. The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: At the internal parameter format converter plug-in, converting the axis data in the time-based metric signal format into internal parameter format data; and The axis data is written to a database.

18. The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: A parameter metric is calculated, wherein the parameter metric is based on one or more of a parameter performance metric or a parameter coverage metric, wherein the parameter metric is based on a usage example.

19. The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: retrieving one or more axis values ​​for the test modality; and One or more axis-specific metrics are calculated based on the one or more axis values.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: The test configuration object is generated based on one or more axis-specific metrics, wherein the one or more axis-specific metrics are based on one or more axis values, and the one or more axis values ​​are selected based on the parameter metric.

Images