Generating simulated environment for testing autonomous vehicle behavior

Abstract

A computer system for generating a scene to run in a simulated environment to test behavior of an autonomous vehicle, the computer system comprising: a rendering component configured to: generate display data to cause a display to render a graphical user interface comprising an image of a driving environment and one or more subjects within the driving environment; a parameter generator configured to generate a user-defined parameter set in the memory in response to a user input defining the parameter set; and an expression manager configured to store a set of user-defined expressions in the memory in response to a user input defining the set of expressions, where each expression of the set of expressions is a user-defined function of one or more parameters of the set of parameters; and a scene generator configured to record a scene of the scene database; wherein the graphical user interface is configured to provide a plurality of subject fields for controlling behavior of one or more subjects when the scene is run in the simulated environment, where each subject field is modifiable to associate it with a parameter of a user-defined parameter set or an expression of a user-defined expression set; the recorded scene includes a driving environment, one or more subjects, a set of user-defined parameters, a set of user-defined expressions, and any user-defined associations (i) between a plurality of subject fields and the set of user-defined parameters and (ii) between a plurality of subject fields and the set of user-defined expressions, wherein each parameter associated with the subject field is controllable to directly modify the subject behavior, and each parameter included in the expression associated with the subject field is controllable to indirectly modify the subject behavior.

Description

Technical Field

[0001] The present disclosure relates to the generation of scenarios for use in a simulation environment for testing the behavior of an autonomous vehicle. Background Art

[0002] There has been significant and rapid development in the field of autonomous vehicles. An autonomous vehicle is a vehicle equipped with sensors and control systems that enable it to operate without human control over its behavior. Autonomous vehicles are equipped with sensors that enable them to perceive their physical environment, such sensors include, for example, cameras, radar, and lidar. Autonomous vehicles are equipped with appropriately programmed computers that are able to process the data received from the sensors and make safe and predictable decisions based on the environment perceived by the sensors. There are different aspects to testing the behavior of sensors and control systems on a specific autonomous vehicle or a type of autonomous vehicle.

[0003] Sensor processing can be evaluated in real-world physical facilities. Similarly, the control systems of autonomous vehicles can be tested in the physical world, such as by repeatedly driving known test routes, or by driving human-driven routes to manage unpredictable or unknown environments.

[0004] Physical world testing will continue to be an important factor when testing the ability of autonomous vehicles to make safe and predictable decisions. However, physical world testing is expensive and time consuming. There is an increasing reliance on the use of simulated environments for testing. If testing in simulated environments increases, it is best that such environments reflect real-world scenarios as closely as possible. Autonomous vehicles need to have the facilities to operate in the same variety of environments that human drivers can operate in. Such situations may contain a high degree of unpredictability.

[0005] It is not feasible to test the behavior of an autonomous vehicle in all possible scenarios that it may encounter during its driving lifetime through physical testing. Increasingly, attention is being placed on the creation of simulation environments that can provide such testing to provide confidence that the test results represent the potential real-world behavior of the autonomous vehicle.

[0006] For effective testing in a simulated environment, the autonomous vehicle under test (the ego vehicle) knows its location at any moment in time, understands its environment (based on simulated sensor inputs), and can make safe and predictable decisions about how to navigate its environment to reach a pre-programmed destination.

[0007] The simulated environment needs to be representative of real-world factors that can vary. This may include weather conditions, road type, road structure, road layout, intersection types, etc. This list is not exhaustive as there are many factors that may affect the operation of the ego vehicle.

[0008] The present disclosure addresses specific challenges that may arise when simulating the behavior of actors in a simulated environment that the ego vehicle will operate in. Such actors may be other vehicles, although they may be other actor types, such as pedestrians, animals, bicycles, etc.

[0009] A simulator is a computer program that, when executed by a suitable computer, is capable of developing and testing in simulation vehicle control modules equipped with sensors before building and testing their physical counterparts. A simulator can provide a sensor simulation system that models each type of sensor that an autonomous vehicle may be equipped with. High-fidelity sensor models can provide realistic or sensor-real synthetic sensor data. Other forms of simulation can be implemented without sensor models or with low-fidelity sensors or perception models. The simulator also provides a three-dimensional environment model that reflects the physical environment in which the autonomous vehicle may operate. The 3D environment model defines at least the road network on which the autonomous vehicle will operate, as well as other actors in the environment. In addition to modeling the behavior of the ego vehicle, the behavior of these actors also needs to be modeled.

[0010] Simulators generate test scenarios (or process scenarios that are provided to them). As already explained, it is important for simulators to be able to generate many different scenarios for testing the ego vehicle, and for good reasons. Such scenarios can include different behaviors of agents. The large number of factors involved in each decision to which an autonomous vehicle must respond, and the large number of other requirements imposed on those decisions (such as safety and comfort are two examples), means that it is not feasible to write a scenario for every situation that needs to be tested. However, efforts must be made to enable simulators to efficiently provide as many scenarios as possible, and to ensure that these scenarios are very close to the real world. If the tests done in simulation do not produce outputs that are consistent with the outputs produced in the corresponding physical world environment, then the value of the simulation is greatly reduced.

[0011] Scenarios can be created from live scenarios recorded from real life driving. Such scenarios can be tagged to identify realistic driving paths and used for simulation. The test generation system can create new scenarios, for example by extracting elements (such as road layout and actor behavior) from existing scenarios and combining them with other scenarios. Scenarios can additionally or alternatively be randomly generated.

[0012] However, there is an increasing need to tailor scenarios for specific environments so that specific sets of factors can be generated for testing. It is hoped that such scenarios can define the behavior of the actors. Summary of the invention

[0013] One aspect of the present disclosure addresses these challenges. According to one aspect, a computer system for generating scenarios to be run in a simulated environment to test the behavior of an autonomous vehicle is provided, the computer system comprising:

[0014] a rendering component configured to: generate display data to cause a display to render a graphical user interface including an image of a driving environment and one or more subjects within the driving environment;

[0015] a parameter generator configured to generate a user-defined parameter set in the memory in response to user input defining the parameter set; and

[0016] an expression manager configured to, in response to user input defining the expression set, store in a memory a user-defined expression set, wherein each expression of the expression set is a user-defined function of one or more parameters of the parameter set; and

[0017] a scenario generator configured to record a scenario in a scenario database;

[0018] wherein the graphical user interface is configured to provide a plurality of subject fields for controlling the behavior of one or more subjects when the scenario is run in the simulation environment, wherein each subject field is modifiable to associate it with a parameter of a user-defined parameter set or an expression of a user-defined expression set; and

[0019] The recorded scenario includes a driving environment, one or more subjects, a user-defined parameter set, a user-defined expression set, and any user-defined associations between (i) multiple subject fields and the user-defined parameter set and (ii) multiple subject fields and the user-defined expression set, wherein each parameter associated with the subject field is modifiable to directly modify the subject behavior, and each parameter included in the expression associated with the subject field is modifiable to indirectly modify the subject behavior.

[0020] Expressions allow dependencies to be defined between subjects and / or their behaviors. For example, a first subject field (such as the starting longitudinal position of the first subject along the road) can be associated with a parameter x, while a second subject field (such as the starting longitudinal position of the second subject) can be associated with an expression involving parameter x (e.g., "x-2m").

[0021] In an embodiment, one or more subjects may be rendered in an image according to corresponding parameters and expressions assigned to the plurality of subject fields.

[0022] The parameter generator may be configured to associate each parameter with a user-defined default value in response to user input defining a default value for the parameter.

[0023] The graphical user interface may be configured to display calculated default parameters for each expression calculated based on user-defined default values ​​for one or more parameters of the expression.

[0024] Alternatively or additionally, one or more subjects may be rendered in the image based on: user-defined default values ​​assigned to parameters of subject fields, and / or default values ​​of expressions assigned to subject fields calculated based on user-defined default values ​​of one or more parameters of the expression.

[0025] The image can be updated when an expression is defined. For example, parameter x can be assigned a default value of 7m by the user, and subjects 1 and 2 can be displayed at longitudinal positions 7m and 7m-2m=5m (the default value of the expression) in the image. These positions can then be automatically updated when the expression changes and / or the default parameter values ​​change. For example, if the default parameter value changes from 7m to 8m, the position of subject 1 becomes 8m, the position of subject 2 becomes 8m-2m=6m, and the image can be updated accordingly.

[0026] When the simulation is subsequently run, different values ​​for the parameters may be sampled (e.g., randomly / using Monte Carlo sampling, uniformly, via a directed search of parameter space, etc.), which in turn are used to evaluate the expression. For example, in the first instance of this scenario, a value of 3m may be sampled for x, so that the starting positions of agents 1 and 2 are 3m and 1m, respectively. In the second instance, a value of 9m may be sampled for x, so that the starting positions of agents 1 and 2 are 9m and 7m, respectively. This allows variation in a way that can be intuitively seen at the design stage, while maintaining the desired relationship between the agent positions.

[0027] In an embodiment, a graphical user interface may include a graphical expression calculator having an expression field for displaying an editable expression, wherein the expression is editable by providing user input representing an expression element to be inserted into the expression. For example, the expression field may be editable by providing user input representing one of the following expression elements to be inserted into the expression displayed in the expression field: a parameter of a user-defined parameter set, a numeric value, a mathematical operator, or a pair of parentheses.

[0028] Expression elements can be inserted at set positions in editable locations, which are not modifiable by the user.

[0029] An expression field can be non-editable and the expression can only be modified by inserting an expression element at a set position, providing a reverse input to remove the most recently inserted expression element (e.g., only the most recently inserted expression or the N most recently inserted expressions), or providing a clear input to clear the expression field.

[0030] For example, in the described embodiments, the expression is editable, but the manner in which it can be edited is intentionally limited. In particular, in the described embodiments, the expression field itself is not editable (the user cannot freely select / modify any part of the expression in the expression field); instead, the user can only insert expression elements at fixed locations (usually at the end of the current expression, or within a pair of "open" brackets at the end of the expression), or remove the most recently inserted expression element (or clear the expression field entirely). These restrictions are imposed to ensure that the expression entered is valid, which in turn helps to ensure that the resulting scenario is valid.

[0031] The graphical expression calculator may include a plurality of parameter elements, each parameter element corresponding to a user-defined parameter and selectable to insert the corresponding parameter into the expression displayed in the expression field.

[0032] The expression may be editable to insert a pair of brackets for receiving a single valid entry, the pair of brackets automatically closing in response to receiving the single valid entry, the single valid entry being a combination of expression elements that satisfy the validity condition.

[0033] A graphical expression evaluator may include a single bracket element that optionally inserts a pair of brackets into an expression.

[0034] At least one subject field may be assigned an expression involving at least one parameter, and the subject may be displayed in the image of the driving environment based on the expression assigned to the at least one subject field and a default value assigned to the at least one parameter.

[0035] At least one of the subject fields may be assigned a parameter, and depending on a default value of the parameter, the subject may be displayed in the image.

[0036] The graphical expression calculator may display, in association with each selectable parameter element of the graphical expression calculator, a user-defined default value for the corresponding parameter.

[0037] A graphical expression calculator may include a default value for the calculation of an expression that updates as the expression displayed in the expression field is edited.

[0038] The recorded scene may include user-defined default values ​​for each parameter of the user-defined parameter set.

[0039] The visualization component may be configured to create one or more subjects in the image of the environment in response to user input for marking one or more locations in the image.

[0040] The parameter generator may be configured to associate each parameter with a user-defined range in response to user input defining a default range for the parameter, wherein the recorded scene may include the user-defined range for each parameter of the user-defined parameter set.

[0041] In order to test the performance of a robotic system in a simulation, the computer system may include: a simulator configured to run one or more instances of a scenario in a simulation environment with the robotic system controlling a self-agent of the scenario; a test oracle configured to process each instance of a test scenario and provide one or more outputs for evaluating the performance of the robotic system therein; a test coordination component, wherein a user-defined parameter set of the scenario is exposed to the test coordination component, and the test coordination component is configured to assign a value to each parameter of the user-defined parameter set for each instance of the scenario, wherein a user-defined expression set is not exposed to the test coordination component, and the simulator is configured to calculate each expression based on the values ​​of one or more parameters assigned to the simulator by the test coordination component.

[0042] The test coordination component may be configured to sample the value of each parameter from its user-defined range.

[0043] Other aspects provide the above-mentioned functions implemented as method steps, and a computer program or a set of computer programs for implementing the functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] For a better understanding of the invention and to show how it may be carried into effect, reference will now be made, by way of example, to the accompanying drawings.

[0045] Figure 1 A highly schematic diagram showing how attributes can be assigned to subjects in a scene is shown.

[0046] Figure 2 An exemplary structure illustrating parameters and a highly schematic diagram of the scene visualizer window are shown.

[0047] Figure 3 A highly schematic diagram illustrating the input of parameters to the expression evaluator is shown.

[0048] Figure 4 The "storyboard" page of the scenario configuration user interface is shown, which allows user configuration of scenarios for simulation.

[0049] Figure 5An exemplary parameter definition page of a scenario configuration user interface is shown.

[0050] Figure 6 An exemplary expression definition page of a scenario configuration user interface is shown.

[0051] Figure 7 An example of a storyboard page of a scene configuration user interface is shown, where expressions are called out and assigned to variables.

[0052] Figure 8 An example of a storyboard page of a scene configuration user interface is shown, wherein a drop-down menu is provided for selecting a parameter or expression to be assigned to a variable.

[0053] Fig. 9A A schematic functional block diagram of an autonomous vehicle stack is shown.

[0054] Fig. 9B A schematic diagram showing an autonomous vehicle testing paradigm.

[0055] Fig.10 A schematic block diagram of a test pipeline is shown.

[0056] Fig.11 Shows the functional block diagram of the scenario editor. DETAILED DESCRIPTION

[0057] The preferred embodiments will now be described by way of example only.

[0058] In order to conduct simulation-based AV performance testing, it is necessary to define scenarios that can be used to test the behavior of the ego vehicle in a simulated environment. Scenarios are defined and edited in an offline mode, where the ego vehicle is not controlled, and then exported for testing in the next stage of the test pipeline 7200 as described below.

[0059] In the described example, the scene includes one or more subjects (sometimes referred to as actors) traveling along one or more paths in a road layout (or more generally, a driving environment). The road layout is a term used in this article to describe any feature that may appear in a driving scene, and in particular, includes at least one track that the vehicle intends to travel along in the simulation. The track can be a road or a lane or any other drivable path. The road layout is displayed in the scene to be edited as an image, and the subject is instantiated on the image. The position in the image is selectable so as to create subjects at these positions. The road layout or other scene topology can be accessed from a database of scene topology. The road layout has lanes, etc. defined therein and presented in the scene. The scene is observed from the perspective of the self-vehicle running in the scene. Other subjects in the scene may include non-self-vehicles or other road users, such as cyclists and pedestrians. The scene may include one or more road features, such as roundabouts or intersections. These subjects are intended to represent real-world entities encountered by the self-vehicle in real-life driving situations. The described system allows users to generate interactions between these subjects and the self-vehicle, which can be executed in the scene editor and then simulated.

[0060] The present specification relates to methods and systems for generating scenarios to obtain large validation sets for testing ego vehicles. The scenario generation scheme described herein enables scenarios to be parameterized and explored in a more user-friendly manner, and also enables scenarios to be reused in a closed loop.

[0061] One aim is to ensure that the scene remains "salient" under different parameterizations (e.g., to ensure that an insertion event actually occurs in many different versions of the scene described by different combinations of parameters); this can be achieved, in part, by using expressions to define appropriate relationships between agents. For example, with independently parameterized agent speeds, it may be the case that a second agent should approach a first agent and insert behind it, but never does so because a particular combination of parameter values ​​means that the second agent's speed is lower than that of the first, such that in a given instance of the scene, the second agent never reaches the first agent; using expressions, this can be avoided by using an expression to relate the second agent's speed to the speed (or target speed) of the first agent, e.g., assigning a parameter v to control the speed (or target speed) of the first agent and relating it to the speed (or target speed) of the second agent via the expression "v+1m / s" (or some other defined function of v), thereby ensuring that the second agent always moves faster (or attempts to move faster) than the first agent, no matter what value v is sampled for.

[0062] In the present system, a scene is described as a set of interactions. Each interaction is defined relatively between an actor of the scene and the static topology of the scene. Each scene may include a static layer for rendering static objects in the environment visualization presented to the user on a display, and a dynamic layer for controlling the movement of moving subjects in the environment. Note that the terms "subject" and "actor" are used interchangeably in this article.

[0063] Each interaction is described relative to an actor and a static topology. Note that in this case, the ego vehicle can be considered a dynamic actor. An interaction consists of an action or behavior performed relative to another actor or a static topology.

[0064] In this article, the term "behavior" can be explained as follows. Behaviors have entities (such as agents in a scene). Given a higher-level goal, behaviors produce interactive actions that move entities toward the given goal. For example, an agent in a scene can be given a lane-following goal and an appropriate behavior model. The agent will (in the scene generated in the editor, and in the resulting simulation) try to achieve that goal.

[0065] Behaviors can be thought of as an opaque abstraction that allows users to inject intelligence into scenes, resulting in more realistic scenarios. By defining a scene as a set of interactions, our system enables multiple actors to work together through proactive behaviors to create a closed-loop behavior network similar to a traffic model.

[0066] The term "action" in this context may be considered as a specific actual action that an entity may perform in order to achieve its specific goal by following its behavior pattern.

[0067] Interactions include conditions and specific actions (or a set of actions) / behaviors with goals that occur relative to two or more actors and / or between an actor and a static scene.

[0068] According to the features of the present system, interactions can be evaluated after the fact using temporal logic. Interactions can be viewed as reusable logic blocks for sequencing scenarios, as described more fully herein.

[0069] Using the concept of interactions, it is possible to define a "critical path" of interactions that are important for a particular scenario. A scenario can have a whole range of abstractions for which parameters can be defined. Variants of these abstract scenarios are called scenario instances.

[0070] Scenario parameters are very important for defining a scenario or an interaction in a scenario. The present system is able to parameterize any scenario value. In the case of expected values ​​in a scenario, parameters can be defined with compatible parameter types and appropriate constraints, as discussed further in this article when describing interactions. As mentioned above, the goal is to facilitate the design of scenarios that remain salient when their parameters are varied in testing (through the use of expressions).

[0071] Scenes are coded in a scene description language (SDL), such as OpenSCENARIO or a custom SDL.

[0072] The scene can be initially created and visualized using paths and bodies using the graph editor tool described in our co-pending application WO2021244956A1, which is incorporated herein by reference in its entirety.

[0073] At test time, one or more parameters of the scenario are exposed, in the sense that different values ​​of the parameters can be chosen to run variations of the scenario. Conceptually, a scenario has a parameter space of one or more dimensions, where each point corresponds to a set of parameter values. Parameter values ​​can be chosen randomly (e.g., by Monte Carlo sampling), or a uniform "grid" of parameter values ​​can be chosen in the parameter space. It is also possible to perform a structured search of the parameter space, called directed testing or directed exploration, where the goal is to find the most significant parameter combination (according to some defined metric) in as few simulation runs as possible, using knowledge gained in earlier simulations. In this case, the scenario parameters are exposed to system components configured to implement a structured search approach.

[0074] For example, refer to the 2021 review paper entitled "A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems" (Corso et al.), which outlines various directed search methods applicable to autonomous cyber-physical systems. This approach relies on scenarios parameterized in a logical manner, preferably capturing all salient aspects of the scenario with as few parameters as possible. The efficiency and quality of the search generally depends on the number of parameters exposed and the relationships between these parameters.

[0075] This description introduces the additional concept of "expressions" that are related to parameters but are not (directly) exposed when testing. An expression is defined as a function of a parameter (or multiple parameters), and the scenario designer is able to map scenario variables (such as subject velocity, position) to parameters or expressions, which in turn affects how the variable is explored when testing. A graphical user interface (GUI) is provided, where subject variables are exposed as editable fields in the GUI, where the fields can be used to map the variables to the desired parameters or expressions.

[0076] The following description relates to the parameterization of the physical and dynamic characteristics and behaviors of the agents in the scene. The parameterization of the scene topology and road layout will be described in more detail below, and further details are provided in our co-pending PCT application PCT / EP2022 / 052123, which is incorporated herein by reference in its entirety.

[0077] Figure 1 Shows a highly schematic block diagram that demonstrates how to use Variables, Parameters, and Expressions to configure agent behavior. Figure 1 Three exemplary subjects are shown: 'Subject 1' 101a, 'Subject 2' 101b and 'Subject 3' 101c, where 'Subject 1' 101a and 'Subject 2' 101b belong to a first subject type 103a, and 'Subject 3' 101c belongs to a second subject type 103b. The assignment of subject type 103 to subjects 101 is determined by Figure 1 The arrow between the body 101 and the body type 103 of the embodiment of the present invention is shown. A user can create parameters in a customized parameter set and expressions in a customized expression set. An expression can be constructed as a user-defined function of one or more parameters of an expression set in the manner described below.

[0078] Notice, Figure 1 Each block in the data structure 103 may represent a data structure. It should be understood that the subject type data structure 103 may define a number of physical properties and / or dynamic limitations of the corresponding subject type, such as size, weight, and maximum acceleration. A subject 101 assigned a particular subject type may inherit the properties, behaviors, and limitations defined in the corresponding subject type data structure 103. In other words, the subject type data structure 103 may define a physical and / or dynamic template of the subject type, on which the subject 101 may be modeled for simulation.

[0079] Each subject data structure 101 may be associated with one or more variables 105. A variable 105 is an abstract data field associated with a particular subject data structure 101 that may be populated to define the dynamic behavior of the corresponding subject during simulation. Figure 1 In the example of FIG. 1 , 'subject 1' 101a is associated with three variables 105a, 105b and 105c. Such variables 105a, 105b, 105c are exposed to the scenario designer via corresponding fields on the scenario editor interface during the design phase.

[0080] Figure 1 Also shown are parameter data store 107 and expression data store 109. Parameter data store 107 may include one or more user-defined parameters, and / or one or more predefined parameters. Figure 2, the parameters 201 may include a default value 203 and a range 205, wherein the range may be user-defined or defined by the computing device so that only significant parameter values ​​are sampled during the simulation. Note that each parameter 201 in the parameter data store 107 may also include data such as a unit of measure, e.g., meters per second or kilograms, etc. This will be described in more detail below. Figure 2 As described above, parameters are exposed at test time (after the scenario design has been completed) to expose the robotic system under test to different variations of the scenario.

[0081] Back to Figure 1 , the parameter data store 107 is shown as including five parameters: 'A', 'B', 'C', 'D', and 'E'. The parameters in the parameter store 107 may be called to populate one or more variables 105 of one or more subjects 101. For example, in Figure 1 Parameter 'A' is called to fill variable 105a of 'body 1' 101a, parameter 'B' is called to fill variable 105b, parameter 'E' is called to fill variable 105c, and parameter 'D' is called to fill variable 105d of 'body 2' 101b. Note that the parameter calls that directly fill body variables 105 Figure 1 This is represented by arrows from the parameters in parameter data store 107 to the corresponding variable block 105 .

[0082] Expressions may also be defined. An expression may be a mathematical expression and may be defined by calling one or more parameters. That is, an expression stored in expression data store 109 may be defined based on one or more parameters stored in parameter data store 107. For example, expression 'F' is defined using parameter 'A' in parameter data store 107. Also, expression 'G' is defined based on parameters 'B' and 'C'. Note that calling parameters to define an expression is not a simple operation. Figure 1 101c and 105e.

[0083] Note that parameters and expressions can be defined by user input to a tool in a user interface (UI) of a computing device, as described in more detail later herein. When defining a parameter, the user can also assign a unit of measure to the defined parameter through the user interface. Parameters can also be defined as scalars.

[0084] Parameters may be called more than once and used for one or more purposes. For example, parameter 'A' is called to define variable 105a, and parameter 'A' is also used in the definition of expression 'F'. In addition, a parameter or variable may be called more than once to populate more than one respective variable 105. In some embodiments, an expression may be defined in terms of another expression, in which case a particular expression may be used more than once and for more than one purpose; for example, the expression may be called in the definition of a second expression, and may also be called to define a variable.

[0085] The expression calculator (a user interface tool for defining expressions) may not provide a tool that allows the input of quantities with units in expressions. In such an example, the user may be required to define parameters with desired units instead, and then call the parameters in the definition of the expression. Alternatively, parameters and expressions may be defined independently of units, and units may be attributed to variables instead. In other words, parameters and expressions may be dimensionless constants that may be assigned to variables associated with units. In order to infer the units of the expression obtained, it is not necessary to use parameters with units. For example, the result expression is a "double" (floating point) type, which can only be applied to any field / variable that is also declared as double, regardless of the unit type of the variable.

[0086] Figure 2 A highly schematic diagram of parameter data store 107 including exemplary parameters 201a and 201b is shown. Figure 2 Also shown is a visualizer window 207 in which images of two bodies 209a and 209b are shown on a road layout 211 .

[0087] Figure 2 The parameter data store 107 shows the parameter structure therein, with each parameter having a default value 203 and a range 205. As previously described, the range can define the upper and lower limits of the parameter values ​​that can be sampled in a simulation run. The configuration of the road layout 211 and the bodies 209a and 209b shown in the visualizer window 207 can indicate the default values ​​of the parameters 201 in the parameter data store 107. Figure 2 Arrow 213 is shown representing one or more steps performed between defining the parameters and viewing the visual rendering of the subject on the road layout, depending on the default values ​​203 of the parameters 201 in the parameter data store 107 .

[0088] Note that the one or more steps represented by arrow 213 may include defining one or more expressions based on one or more parameters 201 in parameter data store 107 and assigning the parameters and expressions to variables associated with subjects in the scene.

[0089] Reference now Figure 3 , which shows a highly schematic diagram of an expression calculator 301 that can be configured to receive user input of parameters 201 to define an expression. As described later herein, a UI can be provided on a computing device that allows a user to provide input and define an expression. Note that when implemented as a UI feature, the expression calculator 301 can include a plurality of selectable buttons corresponding to mathematical operations that can be used to define an expression. Selectable buttons can also be provided for entering values ​​and calling parameters 201 from the parameter data store 107. Figure 3 The input of two exemplary parameters 201 to the expression calculator 301 is shown with respective default values ​​203. Based on the mathematical expression defined in the expression calculator 301, a default expression value 305 is calculated. The default expression value 305 may be calculated by evaluating the mathematical expression defined in the expression calculator 301 using the default values ​​203 of the parameters 201 that appear in the mathematical expression.

[0090] The expression calculator 301 also includes an instance tab 621, which may include a list of variables to which the currently edited expression is assigned. Figure 5 and Figure 6 ,Other examples of instance labels will be described in more detail later.

[0091] The range of values ​​to which an expression can be evaluated depends on the ranges of the individual parameters used to define the expression. Note, however, that the upper and lower bounds of an expression are not necessarily evaluated when the corresponding upper and lower bounds of the constituent parameters are entered into the expression.

[0092] It is desirable for users to be able to set fields in an action based on calculations performed with existing actions. There is a shared overlap between parameters and expressions, as they are both elements that users can map to fields in the GUI.

[0093] Expressions are built in memory in the form of a tree data structure, where each node contains an expression consisting of a single operator (the enumeration is limited to +, -, *, / , ^) and two operands (parameters, int / double values, or expressions). For example, consider the following expression and its tree representation: (x*10)+(y-4) with parameters x and y

[0094] ·expression

[0095] Operator: +

[0096] ·Operation 1:

[0097] ·expression

[0098] Operator: *

[0099] Operand 1: x

[0100] ·Operation 2: 10

[0101] ·Operation 2:

[0102] ·expression

[0103] Operator: -

[0104] Operand 1: y

[0105] ·Operation 2: 4

[0106] In the described implementation, expressions cannot be nested within other expressions (nested expressions are possible in other implementations).

[0107] Figure 4-Figure 8 A series of exemplary UI tools are shown, which a user can use to define one or more subjects and assign dynamic behaviors and physical properties to one or more subjects for simulation. Subject behaviors and properties are assigned by defining parameters and expressions and calling these parameters and expressions to fill in variables associated with one or more subjects.

[0108] Figure 4 An exemplary "storyboard" page 409 of UI 401 is shown. The "storyboard" page 409 of UI 401 can be used to define one or more entities and their characteristics and behaviors for simulation. Figure 4 In the example of , an "Initialize" tab 403 of UI 402 is selected, which provides tools that enable a user to assign physical characteristics (such as predefined subject types) and / or dynamic behaviors (such as routes to be followed by the subjects during simulation) to each of one or more defined subjects. Note that the scene topology and road layout on which the subjects are to be simulated may be predefined, for example from a map database including template road layouts, or may be user defined using a UI tool. The UI tool may be as described in our above-mentioned co-pending application PCT / EP2022 / 052123.

[0109] Note that static entities such as traffic lights and road signs can also be defined via the “initialize” tag 403 .

[0110] Figure 4 The UI also includes a visualizer window 207 in which a graphical representation (image) of the scene defined in the UI 401 can be displayed. Figure 2 As described above, the subject configuration in the visualizer window can be based on default values ​​for parameters and expressions defined by the user and assigned to the subject. Other pages and tabs on UI 401 can be used to define parameters and expressions and assign them to subject variables. For example, Figure 4 The UI 401 shows a first story tag 411 , which may correspond to one or more of the subjects in the scene.

[0111] As described below, agents can be parameterized so that the behavior of a particular agent depends on the behavior of another agent. The behavior of a particular agent can also be linked to the behavior of another agent through the start and end conditions of the respective behaviors being mutually dependent. Story tags 411 can define independent parts of a scene. Story tags can include start conditions and end conditions, and can define one or more agents and one or more corresponding actions for each agent. Story tags define conditional links between each action of each agent, because actions are executed when the start condition is met and stop after the end condition.

[0112] Figure 5 A parameter definition page 501 displayed on the UI 401 is shown. Figure 5 The parameter page 501 includes an optional parameter creation button 503, which, when selected, allows the user to name and define the value of the parameter 201. Figure 5 In the example of , multiple parameters are shown in a list on the parameter page 501 , and each parameter 201 has a corresponding entry in the list.

[0113] Each entry in the list includes a parameter identifier 513. Figure 7 As described in more detail, a parameter 201 may be assigned to a variable 105 by referencing a corresponding parameter identifier 513 in a field configured to define the variable. Thus, each parameter identifier 513 is associated with a data structure address at which data for the corresponding parameter 201 is stored. During simulation, or in a visualizer window representation of a scene, data corresponding to the referenced parameter 201 may be retrieved by accessing the data stored at the address associated with the referenced parameter identifier 513. Note that the data stored for each parameter may include a default value, a parameter value range, a unit, and a corresponding parameter identifier.

[0114] Parameter 201c with parameter reference 513 'Actor_spd' in Figure 5 The parameter page 501 shows an expanded view of the list entry of the 'Actor_spd' parameter 201c, the expanded view including a plurality of fields 505, 507, 509a, 509b and 511 configured to receive user input to define or edit aspects of the corresponding parameter 201c.

[0115] A unit field 505 is provided on the UI 401, and the user can use the unit field 505 to define the unit of measurement for the corresponding parameter 201c. The unit field 505 can be a selectable feature of the UI 401, and when selected, the selectable feature causes a drop-down menu to be displayed on the UI 401, and the selectable menu includes a plurality of predefined units of measurement, from which the user can select an appropriate option. However, as previously mentioned, a parameter can be defined as a dimensionless constant without a unit.

[0116] exist Figure 5 A default field 507 is provided on the UI 401 of , and the default field 507 is configured to receive user input defining a default value 203 of the associated parameter 201c. Figure 5 In the example of , the default value 203 of the parameter 201c is '20.000'. A minimum value field 509a and a maximum value field 509b are also provided. Fields 509a and 509b are configured to receive user input defining the minimum and maximum values, respectively, that the associated parameter 201c can take during the simulation. Figure 5 In the example of , the minimum and maximum values ​​are '15.000' and '27.000' respectively. The values ​​entered into fields 509a and 509b define the lower and upper limits of range 205 associated with parameter 201c, respectively. In examples where the range is computer defined, for example to ensure significance, the range may be determined based on other aspects of the simulation, such as other subject behavior or road layout, etc.

[0117] The parameter definition page 501 also includes a parameter instance field 511. The instance field 511 shows a list of expressions and / or variables that call the associated parameter 201c. The instance field 511 can remain empty unless there is another variable that calls the parameter 201c, or one or more expressions that call the parameter 201c are defined.

[0118] Reference now Figure 6 , which shows an exemplary expression page 601 provided on the user interface 401. The expression page 601 includes a list containing a plurality of entries, each entry in the list corresponding to an expression 603 stored in the expression data store 109.

[0119] Each entry in the list also includes an expression identifier 607. As briefly described above, and as referenced below Figure 7As discussed in detail, an expression 603 may be assigned to a variable 105 by referencing a corresponding expression identifier 607 in a field configured to define the variable 105. Thus, each expression identifier 607 is associated with a data structure address at which data for the corresponding expression 603 is stored. During simulation, or in a visualizer window representation of a scene, default values, ranges, and other data, such as units, associated with a referenced expression 603 may be retrieved. Retrieval of expression data is accomplished by accessing the data structure address associated with the corresponding expression identifier 607.

[0120] The expression page 601 includes a "new expression" feature 605 configured to create a new expression. User selection of the "new expression" feature 605 on the UI 401 may allow the user to enter a new expression identifier 607 and a mathematical function that defines the new expression 603. When the user creates a new expression by selecting the new expression feature 605, a new entry in the list of expressions may be created in an "expanded view". The expanded view of the expression entry in the list may include a UI tool that is configured to receive user input to define the new expression 603. It may also be expanded Figure 6 An existing expression entry in the displayed list is displayed so that the identifier and / or mathematical function that defines the existing expression can be edited.

[0121] exist Figure 6 In the example of , the entry in the expression list corresponds to a first expression 603a with an expression identifier 607 'Exec_time', shown in an expanded view of the expression page 601. UI tools for editing the expression 603a are also provided on the UI. Note that a second exemplary expression 603b with an expression identifier 'Exec_time_follow' is shown in a collapsed view, where no expression editing tools are displayed.

[0122] The tools by which a user can edit an expression 603 (or define a new expression 603) are now described. The calculator input interface 609 is provided with a corresponding calculator display area 611. The calculator input interface 609 includes a plurality of user-selectable buttons that enable the user to enter mathematical functions that define the corresponding expression 603. Decimal points, parentheses, numbers in the range of 0-9, and basic mathematical operator symbols (e.g., addition, subtraction, multiplication, division, or exponential operations) can all have corresponding selectable buttons provided on the calculator input interface 609.

[0123] When a particular selectable button on the calculator input interface 609 is selected, the corresponding number or symbol may be displayed in the display area 611. A "back" button 615 and a "clear" button 617 are also provided on the calculator input interface 609. The back button 615 allows the user to delete (undo) the most recent input to the calculator, thereby removing the input from the display area 611. The clear button 617 may be selected to remove all inputs, thereby removing all numbers and symbols from the display area 611. If the user wishes to undo a previous input, they must first undo all other inputs thereafter.

[0124] Although the user input device of the expression evaluator is not Figure 6 609, it should be understood that a keyboard device may be used to interact with input interface 609. Each selectable button on calculator interface 609 may have a designated "hotkey", i.e., a corresponding button on a physical keyboard device, wherein use of a particular hotkey on the keyboard device may result in a corresponding input to calculator interface 609 as if the corresponding selectable button on interface 609 had been selected.

[0125] Keyboard entry is strictly a "hotkey" type of interaction (as opposed to simply typing into expression display 611, the user cannot directly edit expression display 611 using the keyboard). One benefit of this particular calculator-like interface is that it ensures that the expressions generated are valid (no typographical errors or malformed math).

[0126] Figure 6 The expanded view of the entries in the list of expressions 603 also includes a parameter index 613. The parameter index is shown as a table including a plurality of rows, each row in the parameter index 613 corresponding to a parameter in the parameter store 107 and displaying a parameter identifier 513 and a default value for the parameter.

[0127] Each row in parameter index 613 may be a selectable feature of UI 401 that, when selected, may invoke an associated parameter 201 in expression 603. That is, selection of a particular parameter row in parameter index 613 may indicate a corresponding address in parameter storage 107 from which parameter values ​​may be retrieved to evaluate expression 603. On expression page 601 of UI 401, selection of a particular parameter row in parameter index 613 may cause display of the corresponding parameter identifier 513 in expression calculator display 611. Figure 6 In the example of , an exemplary selectable parameter identifier 513a (parameter button) corresponding to the 'Actor_spd' parameter is displayed in a row of parameter index 613 and is displayed in the function displayed in the display area 611 of the expression calculator. It can therefore be understood that in Figure 6The Exec_time expression 603a being edited in the example of is dependent on the Actor_spd variable; that is, the value of the Exec_time expression 603a depends at least in part on the value taken by the Actor_spd parameter.

[0128] To aid in understanding the results of a calculation, default values ​​are displayed next to the parameters and the expected results of the expression (based on the default values) are displayed on the calculator screen. Thus, the expression page 601 includes an expression default field 619 that displays the default value of the associated expression 603. The default value of the expression may be a numeric output when the expression 603 is evaluated using the corresponding default values ​​of the parameters 201 called out in the expression 603.

[0129] The expression page 601 also includes an expression instance field 621. The instance field 621 shows a list of variables that call the associated expression 603. The instance field 621 can remain empty unless there is another variable that calls the expression 201.

[0130] The user can select the parameters to include by selecting one of the parameter buttons (e.g., 513a). In addition, the user can add operators from the set (+, -, *, / , ^) [addition, subtraction, multiplication, division, and exponentiation] using the corresponding operator button 630. The sub-expression button 632 (bracket or "()" button) can be used to add a new sub-expression.

[0131] When () is pressed, subsequent expression entries are enclosed in parentheses. It automatically closes the parentheses after a single valid entry (e.g., "(a+b)"), that is, a subexpression that satisfies a certain valid condition.

[0132] Specifically, when the () button 632 is selected, a pair of brackets (parentheses) are inserted. Once a valid sub-expression is entered, the sub-expression will automatically be "closed". So, for example, if "()" is pressed, followed by "Actor_spd", "+", and "5", the sub-expression will automatically be closed as "(Actor_SPD+5)"; if the user then presses "*" and "Time_gap", this will be entered as "(Actor_spd+5)*Time_gap" [rather than (Actor_spd+5*Time_gap)], because the sub-expression "Actor_spd+5" is valid. All of these inputs can also be entered using hot keys. Further working examples are listed in the attachment at the end of the specification.

[0133] In order to make the interface more like a traditional calculator, and adhere to the underlying SDL structure, it automatically makes the following decisions.

[0134] Pressing an operator button after an operator symbol in an expression changes that symbol to the key just pressed. For example, if the last symbol in the expression is a "+" and the "-" button is pressed, the "+" will change to a "-".

[0135] Adding a third term to an expression automatically brackets the first two terms; for example, 1+2*3 automatically becomes (1+2)*3.

[0136] The expression calculator input interface 609 does not include selectable function navigation tools. That is, there is no selectable button on the calculator input interface 611 that enables the user to navigate to a specific point in the function shown on the calculator display area 611 without at least deleting the portion of the function after the specific point. The only navigation tools provided on the calculator input interface 609 are the "Back" 615 and "Clear" 617 buttons, which only allow the user to edit a specific point in the function by deleting the content back to that point, or by removing all content and re-entering the function.

[0137] Expressions can be defined with different degrees of complexity. More complex expressions can typically include more parameters or other independent variables, more mathematical operators, and more complex bracket levels. As the complexity of the function defining the expression increases, the difficulty of inputting such a function into the calculator interface 609 with limited navigation tools also increases. For example, inputting a complex function into the calculator interface 609, typically with a complex bracket level, may require a large amount of user effort and / or multiple input attempts before the correct format is entered.

[0138] As mentioned above, the main motivation for this restricted interface is to ensure that users do not create invalid expressions, and these considerations apply to expressions of any complexity. This design is a usability tradeoff for scene completeness. In other embodiments, the user interface can be extended with enhancements to support the creation of more complex expressions.

[0139] The images in the visualizer window 207 are updated when field assignments change (e.g., a parameter assigned to a field is replaced with another parameter or expression in the field, or an expression assigned to a field is replaced with a parameter or another expression). In particular, subject position (location and / or orientation) variables can be visualized based on the fields / parameters to which they are assigned. Movement variables can also be visualized using appropriate visual mechanisms. Default parameter values ​​and default expression values ​​can be used for visualization, and the visualization is updated when the default value changes (the default value of an expression is changed by changing the default value of one of its constituent parameters).

[0140] For example, a subject with a longitudinal position variable assigned to parameter x (default value is 5m) can be displayed as being 5m along the road at a given time step (e.g., t=0). A second subject with a longitudinal position variable assigned to the expression "x-2m" can be displayed as being 5m-2m=3m along the road at that time step.

[0141] Scene visualization 207 is not necessarily static. For example, a moving image can be displayed, or a series of images representing different time steps of a scene can be displayed (e.g., simultaneously or sequentially), and the image can be updated to display the scene of different time steps. In this case, the default value assigned or calculated for the mobile field can control the image at different time steps. For example, a subject can be displayed in an image (or multiple images) at different time steps defined by the default values ​​of the parameters / expressions assigned to its speed and position variables. In this case, the expression assigned to the mobile field, for example, for defining the mobile relationship between the subjects (e.g., defining the speed of subject 2 according to the speed of subject 1) is visualized in the design stage. More complex expressions can be used to define more complex relationships (e.g., speed according to position and time), which can also be visualized. Alternatively or additionally, other visualization mechanisms (such as arrows representing mobile vectors) can be used to indicate mobile variables.

[0142] Figure 7 An example of a UI 401 showing a storyboard page 403 is shown. Figure 7 In the example of , the first story tag 411 titled "Going_str_actors_story" is selected. The storyboard interface structure is designed under the guidance of the OpenSCENARIO file format (for an overview, see https: / / www.asam.net / index.php?eID=dumpFile&t=f&f=4908&token=ae9d9b44ab9257e817072a653b5d5e98ee0babf8#_storyboard_and_entities, the entire contents of which are incorporated herein by reference).

[0143] Figure 7 Variable configuration block 701a is shown, which is a user interface feature configured to receive user input that assigns a parameter or expression to a variable 105 associated with variable configuration block 701a. Figure 7In the example of , variable configuration block 701a is associated with start condition variable 105g. Start condition variable 105g can define an instance in the scene where a particular event or subject behavior starts. Note that the "point in the scene" that satisfies the start condition can be any suitable spatial, temporal or dynamic condition, such as a point in time, a distance traveled, a target speed, etc. Figure 7 The variable configuration block 701a shows the assignment of parameters to scene condition variables, defining the scene conditions. Figure 7 The variable 105 defined in is defined by only one parameter, but it should be understood that, in general, the variable configuration block 701 may include multiple fields, and multiple aspects of the relevant variables are defined by corresponding multiple constants, parameters and / or expressions. For example, an action or condition may have multiple variables, one or some of which may be assigned to expressions, others to parameters, and still others to constants (all in one variable configuration block).

[0144] A parameter 201 or expression 603 may be assigned to a variable 105 by filling a field 703 in a variable configuration block 701 with the identifier of the parameter or expression. Note that the parameter or expression is stored in the respective data store 107, 109. Figure 7 , field 703 associated with variable 105g is populated with a parameter identifier: 'Exec_time'. In practice, assigning a parameter or expression to a variable specifies a data structure address from which the variable value can be retrieved. For example, the data structure address indicated by the identifier can store a default value for the assigned parameter or expression, using which the system can generate a representation of the scene in the visualizer window. The referenced data structure address can also include a range for the assigned parameter or expression within which values ​​can be sampled to generate a simulation run.

[0145] Field 703 may be populated manually. For example, field 703 may be configured to receive user input from a keyboard device, the input being an identifier of a parameter or variable to be assigned. Alternatively, Figure 8 As shown, field 703 may be selected to open a drop-down menu.

[0146] Figure 8 Shown with Figure 7 The same example of user interface 401 is shown. Figure 8 , select field 703 to open a drop-down menu 801, which includes a plurality of entries. Each entry in menu 801 corresponds to a parameter or expression and displays an identifier of the parameter or expression to which it corresponds. Each entry in menu 801 may be a selectable option. When a particular entry is selected, field 703 is filled with the identifier shown in the selected entry, and the parameter or expression corresponding to the selected entry is assigned to variable 105g.

[0147] Each entry in menu 801 includes an icon 803 indicating whether the identifier shown in the entry corresponds to a parameter or an expression. A first icon type 803a is displayed on entries in menu 801 that are associated with parameters in a parameter data store. A second icon type 803b is displayed on entries that are associated with expressions in an expression data store.

[0148] Fig.11 A block diagram of a scene editor is shown, which is shown to include a rendering component 1102, a parameter manager 1103, an expression manager 1105, and a scene generator 1108. The rendering component 1102, the expression manager 1103, and the parameter manager 1105 are shown to be coupled to a set of input / output devices 1100, which include at least one display and at least one input device (touch screen, keyboard, mouse, track pad, etc.). The rendering component 1102 renders a GUI, including a visualization. The scene generator is configured to manage scenes, including subjects added to the scene, and enter their variables. The parameter manager 1103 manages parameters stored as parameter sets 1104 and their default values. The expression manager 1105 manages expressions and calculates their default values. The scene generator 1108 maintains the association between subject variables and parameters / expressions, and updates the visualization based on the default values ​​of the parameters / expressions assigned to the variables. The scene generator 1108 is also responsible for recording scenes in the scene database 1070 for simulation use. The recorded scene summarizes the user-defined parameters and the relationships contained in the user-defined expressions. Default parameter values ​​can also be recorded, for example for visualization / preview of rendered scenes elsewhere in the system. It is understood that Fig.11 is highly schematic and does not necessarily reflect the code or program structure of the Scenario Editor. Fig.11 The functional blocks in a usually represent some subset of the scene editor functionality.

[0149] Simulate deployment

[0150] Fig. 9A A highly schematic block diagram of an AV runtime stack 900 is shown by way of context. The runtime stack 900 is shown to include a perception system 902, a prediction system 904, a planning system (planner) 906, and a control system (controller) 908.

[0151] In the real world environment, the perception system 902 receives sensor outputs from the AV's onboard sensor system 910 and uses these sensor outputs to detect external subjects and measure their physical states, such as their position, velocity, acceleration, etc. The onboard sensor system 110 can take different forms, but typically includes a variety of sensors, such as image capture devices (cameras / optical sensors), lidar and / or radar units, satellite positioning sensors (GPS, etc.), motion / inertial sensors (accelerometers, gyroscopes, etc.), etc. Therefore, the onboard sensor system 910 provides a rich sensor data from which detailed information about the surrounding environment can be extracted, as well as the state of the AV and any external actors (vehicles, pedestrians, cyclists, etc.) in the environment. The sensor outputs typically include sensor data of multiple sensor modalities, such as stereo images from one or more stereo optical sensors, lidars, radars, etc. Filters, fusion components, etc. can be used to combine sensor data of multiple sensor modalities. The perception system 902 typically includes multiple perception components that collaborate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 902. In a simulated environment, synthetic sensor data may be fed to the perception system 902, such as generated using a high-fidelity sensor model. Alternatively, the perception system 902 (or one or more portions thereof) may be replaced with a surrogate model operating on lower-fidelity inputs from a simulator.

[0152] The predictions calculated by the prediction system 904 are provided to the planner 906, which uses the predictions to make autonomous driving decisions to be performed by the AV in a given driving scenario. The core function of the planner 906 is to plan the trajectory of the AV (ego trajectory) taking into account the predicted subject movement. The planning trajectory is to achieve the desired goal in the scenario. For example, the goal can be to enter a roundabout and leave at the desired exit, overtake the vehicle in front, or stay in the current lane at a target speed (lane following). The goal can be determined, for example, by an autonomous route planner (not shown). The controller 908 executes the decisions made by the planner 906 by providing appropriate control signals to the AV's on-board actor system 912. In particular, the planner 906 plans the trajectory for the AV, and the controller 908 generates control signals to achieve the planned trajectory.

[0153] Fig. 9AThe example considers a relatively "modular" architecture with separable perception, prediction, planning, and control systems 902-908. The degree of integration or separation of the various stack functions may vary significantly in different stack implementations, and in some stacks, some aspects may be so tightly coupled as to be indistinguishable. In extreme cases, in so-called "end-to-end" driving, perception, prediction, planning, and control may be essentially inseparable. Unless otherwise noted, the terms perception, prediction planning, and control used herein do not imply any particular coupling or modularity of these aspects.

[0154] It should be understood that the term "stack" includes software, but may also include hardware. In simulation, the stack's software may be tested on a "generic" off-board computer system before ultimately being uploaded to the physical vehicle's on-board computer system. However, in "hardware-in-the-loop" testing, testing may extend to the underlying hardware of the vehicle itself. For example, the stack software may be run on an on-board computer system (or a replica thereof) coupled to a simulator for testing purposes. In this case, the stack under test extends to the vehicle's underlying computer hardware. As another example, certain functions of the stack 900 (e.g., perception functions) may be implemented in dedicated hardware. In a simulation environment, hardware-in-the-loop testing may involve feeding synthetic sensor data to dedicated hardware perception components.

[0155] Fig. 9B A highly schematic overview of an autonomous vehicle testing paradigm is shown. The ADS / ADAS stack 900 (e.g., Fig. 9AThe output of the test oracle 1052 is informative to the experts 922 (team or individual), allowing them to identify problems in the stack 900 and modify the stack 900 to mitigate those problems (S124). The results also help the experts 922 select further test scenarios (S126), and the process continues, repeatedly modifying, testing, and evaluating the performance of the stack 900 in simulation. The improved stack 900 is ultimately incorporated (S125) into a real-world AV 901 equipped with a sensor system 910 and an actor system 912. The improved stack 900 typically includes program instructions (software) executed in one or more computer processors of an onboard computer system (not shown) of the vehicle 901. At step S125, the software of the improved stack is uploaded to the AV 901. Step S125 may also include modifications to the underlying vehicle hardware. On the AV 901, the improved stack 900 receives sensor data from the sensor system 910 and outputs control signals to the actor system 912. Real-world testing (S128) can be used in conjunction with simulation-based testing. For example, once an acceptable level of performance has been achieved through the process of simulation testing and stack refinement, appropriate real-world scenarios can be selected (S130), and the performance of the AV 101 in these real-world scenarios can be captured and similarly evaluated in the test oracle 1052.

[0156] Fig.10 A schematic block diagram of a test pipeline 1000 is shown. The test pipeline 1000 is shown to include a simulator 1002 and a test oracle 1052. The simulator 1002 runs simulation scenarios to test the AV runtime stack 900, and the test oracle 1052 evaluates the performance of the stack (or sub-stack) on the simulation scenarios. As discussed, only a portion (or portions) of the runtime stack may be tested, but for simplicity, the following description refers to the (complete) AV stack 1000 throughout.

[0157] As mentioned before, the idea of ​​simulation-based testing is to run a simulated driving scenario 1001 coded in SDL, which the ego agent must navigate under the control of the stack 9000 being tested. The scenario 1001 includes one or more user-defined parameters 1000b and one or more expressions 1000a. The user-defined relationships between the variables of the scenario 1001 (such as the subject's position, direction, speed, acceleration, etc.), expressions 1001a and parameters 1000b are recorded in the scenario 1001.

[0158] The simulated input 1003 is used by the planner 908 (directly or indirectly) as the basis for decision making. As described above, the simulated input 1003 can be high-fidelity synthetic sensor data, or low-fidelity input. For example, a substitute model of all or part of the perception system 902 can be used for testing, operating on low-fidelity input. The controller 908 in turn executes the planner's decisions by outputting control signals 909. In a real-world environment, these control signals will drive the AV's physical actor system 912. In the simulation, the self-vehicle dynamic model 1004 is used to convert the obtained control signals 909 into the real movement of the self-agent in the simulation, thereby simulating the physical response of the autonomous vehicle to the control signals 909. Alternatively, a simpler form of simulation assumes that the self-agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 908 (to some extent, it can be separated from planning) and eliminates the need for the self-vehicle dynamic model 1004. This may be sufficient to test certain aspects of planning. One or more subject dynamic models 1006 can be used to provide realistic subject behavior in simulation.

[0159] The output of the simulator 1002 for a given simulation includes an ego trajectory 1012a for the ego subject and one or more subject trajectories 1012b for one or more external subjects (trajectories 1012). Each trajectory 1012a, 1012b is a complete history of the subject's behavior in the simulation with spatial and motion components. For example, each trajectory 1012a, 1012b can take the form of a spatial path with motion data associated with points along the path, such as velocity, acceleration, jerk (rate of change of acceleration), rapidity (rate of change of jerk), etc. Additional environmental data 1014 about the trajectory is provided.

[0160] The test oracle 1052 receives the trajectory 1012 and the environmental data 1014 and scores these outputs according to a set of performance evaluation rules 1054. The test oracle 1056 provides. The test oracle computes output 1056 representing the performance of the stack 900 relative to the driving rules 1054.

[0161] The scenario 1001 is created using the scenario editor described above and recorded in the scenario database 1070. The scenario database 1070 may contain a large number of recorded scenarios.

[0162] The user-defined parameters 1000b of the scenario 1001 are exposed to a component 1060 (a test coordination component) of the system that is responsible for determining changes to the scenario 1001. For example, the test coordination 1060 component can randomly (e.g., based on Monte Carlo sampling) select values ​​for the parameters 1000b via a uniform grid search, using a directed test / directed exploration method, etc. An instance or "run" of a scenario refers to an instantiation in the simulator with a particular set of parameter values. Thus, the scenario variables with assigned parameters are directly controlled by controlling these parameters (i.e., the sampled values ​​of these parameters).

[0163] Parameter values ​​are sampled according to a user-defined range. For example, parameter values ​​can be selected uniformly or randomly from their user-defined range.

[0164] Expression 1000a is not exposed to test coordination component 1060 in this way. Instead, expression 1000a is evaluated based on sampled parameter values ​​as needed to compute a simulation based on the parameter values ​​sampled in a given run. Thus, scene variables with assigned expressions are indirectly controlled by sampling the parameter values ​​contained in those expressions and using the sampled values ​​to compute the value of the expression. This allows for greater control over the scene, for example, ensuring that two agents maintain a certain distance (even if their starting positions change between runs), or ensuring that events such as cutting in or overtaking occur by relating the speeds of agents to each other (e.g., avoiding a situation where an overtaking agent never reaches the agent in front of it that it should overtake).

[0165] The agent parameters / expressions and their sampled / computed values ​​may, for example, form input to the agent dynamics model 1006 .

[0166] Here, for example Fig.11References to components, functions, modules, etc., such as components 1102, 1103, 1105, and 1108 of the scene editor in the example above represent functional components of a computer system that can be implemented at the hardware level in various ways. The computer system includes execution hardware that can be configured to execute the method / algorithm steps disclosed herein and / or implement a model trained using the present technology. The term execution hardware includes any form / combination of hardware configured to execute the relevant method / algorithm steps. The execution hardware can take the form of one or more processors, which can be programmable or non-programmable, or a combination of programmable and non-programmable hardware can be used. Examples of suitable programmable processors include general-purpose processors (such as CPUs, GPU / accelerator processors, etc.) based on an instruction set architecture. Such general-purpose processors typically execute computer-readable instructions stored in a memory coupled to the processor or within the processor, and perform relevant steps according to these instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs), which have circuit configurations programmable by circuit description codes. Examples of non-programmable processors include application-specific integrated circuits (ASICs), codes, instructions, and the like. Storage may suitably be on either transient or non-transitory media (examples of the latter include solid-state, magnetic and optical storage devices, etc.).

[0167] appendix

[0168] Calculation interface working example

[0169] Target (x*20) - 5

[0170] Press the x parameter button

[0171] x

[0172] Press the multiplication button

[0173] x*

[0174] Type 20 using the numeric keypad

[0175] x*20

[0176] Press the subtraction button

[0177] (x*20)-

[0178] Type 5 in the numeric input and press the insert button (x*20)-5

[0179] Target (x*10)+(y^2)

[0180] Press the x parameter button

[0181] x

[0182] Press the multiplication button

[0183] x*

[0184] Type 10 using the numeric keypad

[0185] x*10

[0186] Press the Add button

[0187] (x*10)+

[0188] Press the () button

[0189] (x*10)+()

[0190] Press the y button

[0191] (x*10)+(y)

[0192] Press the ^ button

[0193] (x*10)+(y^)

[0194] Type 2 in the numeric input and press the Insert button

[0195] (x*10)+(y 2 )

Claims

1. A computer system for generating scenarios to be run in a simulated environment to test the behavior of an autonomous vehicle, the computer system comprising: a rendering component configured to: generate display data to cause a display to render a graphical user interface including an image of a driving environment and one or more subjects within the driving environment; a parameter generator configured to generate a user-defined parameter set in a memory in response to user input defining the parameter set; as well as an expression manager configured to store in a memory a user-defined set of expressions in response to user input defining the set of expressions, wherein each expression of the set of expressions is a user-defined function of one or more parameters of the set of parameters; and a scenario generator configured to record a scenario in a scenario database; wherein the graphical user interface is configured to provide a plurality of subject fields for controlling the behavior of the one or more subjects when the scenario is run in a simulation environment, wherein each subject field is modifiable to associate it with a parameter of the user-defined parameter set or an expression of the user-defined expression set; and The recorded scenario includes the driving environment, the one or more subjects, the user-defined parameter set, the user-defined expression set, and any user-defined associations (i) between the multiple subject fields and the user-defined parameter set and (ii) between the multiple subject fields and the user-defined expression set, wherein each parameter associated with the subject field is controllable to directly modify the subject behavior, and each parameter included in the expression associated with the subject field is controllable to indirectly modify the subject behavior.

2. The computer system according to claim 1, wherein: The graphical user interface includes a graphical expression calculator having an expression field for displaying an editable expression, wherein the expression is edited by providing user input representing expression elements to be inserted into the expression.

3. The computer system according to claim 2, wherein: The expression is edited by providing user input representing one of the following expression elements to be inserted into the expression displayed in the expression field: the parameters of the user-defined parameter set, Numeric value, Mathematical operators, or A pair of brackets.

4. The computer system according to claim 2 or 3, wherein: The expression element is inserted into a set position in an editable location that is not user-modifiable.

5. A computer system according to claim 2, 3 or 4, wherein: The expression field is not editable, where the expression can only be modified in the following ways: Insert an expression element at the set position, Provide reverse input to remove the most recently inserted expression element, or A clear input is provided to clear the expression field.

6. The computer system according to any one of claims 2 to 5, wherein: The graphical expression calculator includes a plurality of parameter elements, each parameter element corresponding to the user-defined parameter and selectable to insert the corresponding parameter into the expression displayed in the expression field.

7. The computer system according to any one of claims 2 to 6, wherein: The expression is editable to insert a pair of brackets for receiving a single valid entry, the pair of brackets automatically closing in response to receiving the single valid entry, the single valid entry being a combination of expression elements that satisfy a validity condition.

8. The computer system according to any one of claims 2 to 7, wherein: The graphical expression calculator includes a single bracket element that optionally inserts a pair of brackets into the expression.

9. A computer system according to any preceding claim, wherein: The parameter generator is configured to associate each parameter with a user-defined default value in response to user input defining a default value for the parameter.

10. The computer system according to claim 9, wherein: The graphical user interface is configured to display a calculated default value for each expression calculated based on a default value of a user-defined default value for one or more parameters of the expression.

11. A computer system according to claim 9 or 10 as appended to claim 2, wherein: The graphical expression calculator displays a user-defined default value for a corresponding parameter in association with each selectable parameter element of the graphical expression calculator, wherein the graphical expression calculator includes a calculated default value for the expression, and the calculated default value is updated as the expression displayed in the expression field is edited.

12. A computer system according to claim 9, 10 or 11, wherein: The recorded scenario includes a user-defined default value for each parameter of the user-defined parameter set.

13. The computer system according to any one of claims 9 to 12, wherein: The one or more subjects are rendered in the image based on: User-defined default values ​​assigned to parameters of body fields, and / or A default value assigned to an expression of a subject field, calculated based on user-defined default values ​​for one or more parameters of the expression.

14. A computer system according to any preceding claim, wherein: The visualization component is configured to create the one or more subjects in the image of the environment in response to user input for marking one or more locations in the image.

15. A computer system according to any preceding claim, wherein: The parameter generator is configured to, in response to user input defining a user-defined range for a parameter, associate each parameter with the user-defined range, wherein the recorded scene includes the user-defined range for each parameter of the set of user-defined parameters.

16. A computer system according to any preceding claim, for testing the performance of a robotic system in a simulation, the computer system comprising: a simulator configured to run one or more instances of the scenario in a simulated environment with the robotic system controlling an ego agent of the scenario; a test oracle configured to process each instance of a test scenario and provide one or more outputs for evaluating the performance of the robotic system therein; A test coordination component, wherein a user-defined parameter set for the scenario is exposed to the test coordination component and the test coordination component is configured to assign a value to each parameter of the user-defined parameter set for each instance of the scenario, wherein the user-defined expression set is not exposed to the test coordination component and the simulator is configured to evaluate each expression based on the values ​​assigned to one or more of its parameters by the test coordination component.

17. A computer system according to claim 16 as appended to claim 15, wherein: The test coordination component is configured to sample the value of each parameter from its user-defined range.

18. One or more computer programs for generating scenarios to be run in a simulated environment to test the behavior of an autonomous vehicle, the one or more computer programs being configured to program a computer system to: generating display data to cause a display to render a graphical user interface including an image of a driving environment and one or more subjects within the driving environment; generating a user defined parameter set in a memory in response to user input defining the parameter set; as well as In response to user input defining the expression set, storing the user-defined expression set in a memory, wherein each expression of the expression set is a user-defined function of one or more parameters of the parameter set; and Recording scenarios in a scenario database; wherein the graphical user interface is configured to provide a plurality of subject fields for controlling the behavior of the one or more subjects when the scenario is run in a simulation environment, wherein each subject field is modifiable to associate it with a parameter of the user-defined parameter set or an expression of the user-defined expression set; and The recorded scenario includes the driving environment, the one or more subjects, the user-defined parameter set, the user-defined expression set, and any user-defined associations (i) between the multiple subject fields and the user-defined parameter set and (ii) between the multiple subject fields and the user-defined expression set, wherein each parameter associated with the subject field is controllable to directly modify the subject behavior, and each parameter included in the expression associated with the subject field is controllable to indirectly modify the subject behavior.

Images