Methods and Apparatus

Abstract

A computer-implemented method for generating trajectories for an actor is disclosed, the method including the steps of simulating a first scenario including an environment having an ego-vehicle, a collection of actors including a first actor, and optionally a collection of objects including a first object, the step including using a first trajectory of the first actor; observing a first observation of the environment (e.g., the ego-vehicle, a second actor in the collection, and / or a first object in the collection) by a first adversarial reinforcement learning agent in response to the first trajectory of the first actor; and generating a second trajectory for the first actor by the first agent based on the first observation of the observed environment.

Description

[Technical Field]

[0001] The present invention relates to autonomous vehicles. [Background technology]

[0002] Traditional testing of autonomous vehicle (AV) control software (also known as the AV stack), such as for SAE Level 1 through Level 5, is problematic. For example, traditional testing methods typically involve manual (i.e., human) and labor-intensive procedures.

[0003] 1. Test drive AVs on real-world roads or in simulated environments with randomly occurring traffic, collecting data on the scenarios encountered and AV behavior.

[0004] 2. Identify challenging scenarios based on AV behavior (e.g., scenarios where a safety driver needs to intervene and the AV does not brake early enough).

[0005] 3. Recreate challenging scenarios in simulation and add random noise to scenario parameters (e.g., positions and velocities of surrounding vehicles / pedestrians / cyclists).

[0006] This approach is not only very expensive and time-consuming, but also requires capturing low-probability events that cannot be captured multiple times. While randomization of scenario parameters based on initial scenarios identified through real-world driving can increase the number of scenarios, this is highly inefficient due to the number of miles required to identify these rare extreme-case scenarios. Undetected control software flaws increase the risk to the AV and its occupants.

[0007] Therefore, it is necessary to conduct tests against AV. Summary of the Invention

[0008] According to a first aspect, a computer-implemented method for generating a trajectory of an actor comprises: simulating a first scenario including an environment having an ego-vehicle, a set of actors including a first actor, and optionally a set of objects including a first object, using a first trajectory of the first actor; Observing, by a first adversarial reinforcement learning agent, a first observation of an environment (e.g., an ego-vehicle, a second actor in the set, and / or a first object in the set) in response to a first trajectory of a first actor; generating, by the first agent, a second trajectory for the first actor based on the first observation of the observed environment.

[0009] According to a second aspect, there is provided a computer-implemented method for simulating a scenario, the method comprising: The method includes generating a first trajectory of a first actor of a set of actors according to a first aspect; simulating a first scenario including an environment having an ego-vehicle, a set of actors including the first actor, and optionally a set of objects including a first object, wherein simulating the first scenario includes using the generated first trajectory of the first actor; and identifying defects in the ego-vehicle in the first scenario.

[0010] According to a third aspect, a computer-implemented method for developing an own vehicle includes: simulating a scenario according to the second aspect; and repairing the identified defects of the host vehicle.

[0011] A computer according to a fourth aspect includes a processor and memory configured to carry out the methods according to the first, second and / or third aspects.

[0012] A computer program according to a fifth aspect comprises instructions that, when executed by a computer including a processor and a memory, cause the computer to perform the methods according to the first aspect, the second aspect and / or the third aspect.

[0013] A non-transitory computer-readable storage medium according to a sixth aspect includes instructions that, when executed by a computer including a processor and a memory, cause the computer to perform the methods according to the first aspect, the second aspect, and / or the third aspect.

[0014] According to one aspect of the present disclosure, a computer-implemented method for generating new adversarial scenarios and agents involving autonomous vehicles includes performing reinforcement learning to train an agent using an autonomous vehicle software stack in a reinforcement learning environment to generate one or more episodes, each representing an adversarial scenario that ends in a failure of the autonomous vehicle software stack; generating a plurality of descriptors based on the or each episode; and storing the plurality of descriptors in a database.

[0015] The autonomous vehicle may be the self-driving vehicle. The adversarial scenario may include a fault in the autonomous vehicle software stack. The agent may be a machine learning model. The machine learning model may include a neural network.

[0016] In one embodiment, the computer-implemented method may include clustering a plurality of descriptors for the or each episode, and storing the plurality of descriptors includes storing the clusters of descriptors in a database.

[0017] The computer-implemented method further includes generating new descriptors by moving from clusters of descriptors in the descriptor space.

[0018] Moving from the cluster of descriptors in descriptor space may include identifying a centroid of the cluster, moving a unit amount in a unit direction from the centroid to a new descriptor location, and generating the new descriptor as the descriptor at the new descriptor location.

[0019] Moving from a cluster of descriptors in descriptor space may include identifying an established boundary of the cluster, moving a unit amount in a unit direction from the boundary to a new descriptor location, and generating the new descriptor as the descriptor at the new descriptor location.

[0020] Moving from a cluster of descriptors in descriptor space may include identifying an established boundary of the cluster, moving a unit amount from the boundary in a local normal direction to a new descriptor location, and generating a new descriptor as the descriptor at the new descriptor location.

[0021] The established boundaries may be identified using a signed distance function.

[0022] The one or more episodes may include multiple episodes, clustering the multiple episodes may include generating multiple clusters, storing the clusters may include storing the multiple clusters in a database, and navigating from the clusters may include navigating from the multiple clusters by determining a union between each cluster, determining a difference between the cluster space and the union, determining a centroid of the difference, and generating a new descriptor as a descriptor at the centroid of the difference.

[0023] The computer-implemented method may further include generating a seed state from the new descriptor, and re-performing reinforcement learning using the seed state, generating a plurality of descriptors, and storing the plurality of descriptors.

[0024] The computer-implemented method may further include re-initializing the agent, and re-performing reinforcement learning using the re-initialized agent, generating the plurality of descriptors, and storing the plurality of descriptors.

[0025] The environment may further include context data.

[0026] The context data may include one or more internal maps and / or one or more external maps.

[0027] The computer-implemented method may further include modifying the context data of the environment, and re-performing the reinforcement learning, generating the plurality of descriptors, and storing the plurality of descriptors using the modified context data.

[0028] An episode may include multiple points, each of which may include a state output by the environment and an action output by the agent. These points may be time points or location points for the autonomous vehicle.

[0029] Generating the plurality of descriptors may include encoding each of the plurality of points into a latent space.

[0030] The failure may include an event selected from the group including a collision between the agent and the autonomous vehicle software stack occurring, a distance between the agent and the autonomous vehicle software stack being less than a minimum distance threshold, a deceleration of the autonomous vehicle software stack being greater than a deceleration threshold, an acceleration of the autonomous vehicle software stack being greater than an acceleration threshold, and a jerk of the autonomous vehicle software stack being greater than a jerk threshold.

[0031] According to one aspect of the present disclosure, a computer-implemented method for generating an agent from a scenario including an autonomous vehicle includes performing reinforcement learning to train an agent using an autonomous vehicle software stack in a reinforcement learning environment to generate one or more episodes, each episode representing an adversarial scenario that terminates with a failure of the autonomous vehicle software stack; re-running reinforcement learning on the agent to generate new episodes; comparing the new episodes to the one or more episodes; and generating the agent by replicating the agent trained using reinforcement learning based on the comparison.

[0032] The failure may include an event selected from the group including a collision between the agent and the autonomous vehicle software stack occurring, a distance between the agent and the autonomous vehicle software stack being less than a minimum distance threshold, a deceleration of the autonomous vehicle software stack being greater than a deceleration threshold, an acceleration of the autonomous vehicle software stack being greater than an acceleration threshold, and a jerk of the autonomous vehicle software stack being greater than a jerk threshold.

[0033] The environment may further include context data.

[0034] The context data may include one or more internal maps and / or one or more external maps.

[0035] An episode may include multiple points, each of which includes a state output by the environment and an action output by the agent. These points may be time points or location points for the autonomous vehicle.

[0036] Comparing the new episode to the one or more episodes may include determining a variance between the new episode and the one or more episodes, and generating the agent by replicating the agent trained using reinforcement learning based on the comparison may include replicating the agent trained using reinforcement learning if the variance is less than a variance threshold.

[0037] According to one aspect of the present disclosure, a computer-implemented method for generating new adversarial scenarios and agents involving autonomous vehicles includes performing reinforcement learning to train an agent using a proxy for the autonomous vehicle software stack in a reinforcement learning environment to generate one or more episodes, each representing an adversarial scenario that ends with a failure of the proxy for the autonomous vehicle software stack; generating a plurality of descriptors based on the or each episode; and storing the plurality of descriptors in a database.

[0038] The computer-implemented method may include clustering a plurality of descriptors for the or each episode, and storing the plurality of descriptors may include storing the clusters of descriptors in a database.

[0039] The computer-implemented method may further include generating new descriptors by moving from clusters of descriptors in the descriptor space.

[0040] Moving from the cluster of descriptors in descriptor space may include identifying a centroid of the cluster, moving a unit amount in a unit direction from the centroid to a new descriptor location, and generating the new descriptor as the descriptor at the new descriptor location.

[0041] Moving from a cluster of descriptors in descriptor space may include identifying an established boundary of the cluster, moving a unit amount in a unit direction from the boundary to a new descriptor location, and generating the new descriptor as the descriptor at the new descriptor location.

[0042] Moving from a cluster of descriptors in descriptor space may include identifying an established boundary of the cluster, moving a unit amount from the boundary in a local normal direction to a new descriptor location, and generating a new descriptor as the descriptor at the new descriptor location.

[0043] The established boundaries may be identified using a signed distance function.

[0044] The one or more episodes may include multiple episodes, clustering the multiple episodes may include generating multiple clusters, storing the clusters may include storing the multiple clusters in a database, and navigating from the clusters may include navigating from the multiple clusters by determining a union between each cluster, determining a difference between the cluster space and the union, determining a centroid of the difference, and generating a new descriptor as a descriptor at the centroid of the difference.

[0045] The computer-implemented method may further include generating a seed state from the new descriptor, and re-performing reinforcement learning using the seed state, generating a plurality of descriptors, and storing the plurality of descriptors.

[0046] The computer-implemented method may further include re-initializing the agent, and re-performing reinforcement learning using the re-initialized agent, generating the plurality of descriptors, and storing the plurality of descriptors.

[0047] The environment may further include context data.

[0048] The context data may include one or more internal maps and / or one or more external maps.

[0049] The computer-implemented method may further include modifying the context data of the environment, and re-performing the reinforcement learning, generating the plurality of descriptors, and storing the plurality of descriptors using the modified context data.

[0050] An episode may include multiple points, each of which may include a state output by the environment and an action output by the agent, and the multiple points may be time points or location points for the autonomous vehicle.

[0051] Generating the plurality of descriptors may include encoding each of the plurality of points into a latent space.

[0052] The failure may include an event selected from the group including a collision between the agent and the autonomous vehicle software stack occurring, a distance between the agent and the autonomous vehicle software stack being less than a minimum distance threshold, a deceleration of the autonomous vehicle software stack being greater than a deceleration threshold, an acceleration of the autonomous vehicle software stack being greater than an acceleration threshold, and a jerk of the autonomous vehicle software stack being greater than a jerk threshold.

[0053] The proxy may include a machine learning model, which is optionally a neural network, which is optionally a convolutional neural network.

[0054] According to another aspect, a computer-implemented method for generating an agent from a scenario including an autonomous vehicle includes providing an agent trained using reinforcement learning in an environment with a proxy of an autonomous vehicle software stack, and performing reinforcement learning to optimize the agent using a full autonomous vehicle software stack on which the proxy is based.

[0055] This aspect may optionally be expressed as a computer-implemented method for a new adversarial scenario and agent involving an autonomous vehicle, the method including: providing an agent trained using reinforcement learning in an environment with a proxy of an autonomous vehicle software stack; performing reinforcement learning to optimize the agent using a full autonomous vehicle software stack on which the proxy is based; generating one or more episodes when optimizing the agent; and generating a plurality of descriptors for each of the other episodes.

[0056] The agent may include providing a trained agent when performing the computer-implemented method of the aforementioned aspect.

[0057] According to one aspect of the present disclosure, a computer-implemented method for generating anomalous trajectory data for an agent in a self-driving vehicle scenario includes receiving, by an adversarial machine learning model, context data including non-anomalous trajectory data for the agent; generating, by the adversarial machine learning model, anomalous trajectory data from the context data; and storing the anomalous trajectory data in a database.

[0058] The autonomous vehicle may be the self-driving vehicle.

[0059] The adversarial machine learning model may include a generative adversarial network trained to generate anomalous trajectory data from non-anomalous trajectory data.

[0060] The computer-implemented method may further include receiving the noise with an adversarial machine learning model, and generating anomalous trajectory data from the context data with the adversarial machine learning model includes generating the anomalous trajectory data based on the noise.

[0061] The context data may further include an internal map and / or an external map.

[0062] The non-anomalous trajectory data may include trajectory data associated with no violations between the agent and the autonomous vehicle.

[0063] The anomalous trajectory data may include trajectory data associated with a violation between the agent and the autonomous vehicle, or trajectory data not associated with no violation between the agent and the self-vehicle.

[0064] The violation may include an event selected from the group including: a collision occurring, coming within a minimum distance, a deceleration of the autonomous vehicle exceeding a deceleration threshold, an acceleration of the autonomous vehicle exceeding an acceleration threshold, and a jerk of the autonomous vehicle exceeding a jerk threshold. In other words, the event may be an event selected from the group including: a collision occurring between the agent and the autonomous vehicle software stack, a distance between the agent and the autonomous vehicle software stack being less than a minimum distance threshold, a deceleration of the autonomous vehicle software stack being greater than a deceleration threshold, an acceleration of the autonomous vehicle software stack being greater than an acceleration threshold, and a jerk of the autonomous vehicle software stack being greater than a jerk threshold.

[0065] According to one aspect of the present disclosure, a computer-implemented method for training an adversarial machine learning model to generate anomalous trajectory data includes providing context data including non-anomalous trajectory data of an agent as input to an adversarial machine learning model; generating predicted anomalous trajectory data from the context data with the adversarial machine learning model; calculating a loss between the predicted anomalous trajectory data and the non-anomalous trajectory data; and modifying parameter settings of the adversarial machine learning model to reduce the loss.

[0066] The adversarial machine learning model may include a generative adversarial network.

[0067] The generative adversarial network may be a first generative adversarial network that forms part of a cycle generative adversarial network that includes a second generative adversarial network. The method may include providing the generated anomalous trajectory data as input to the second generative adversarial network, generating reconstructed non-anomalous trajectory data with the second generative adversarial network, calculating a loss between the reconstructed non-anomalous trajectory data and the non-anomalous trajectory data, and modifying parameter settings of the second generative adversarial network to reduce the second loss, where the loss is the first loss.

[0068] The second loss may include a reconstruction loss and / or an adversarial loss.

[0069] The losses may include adversarial losses and / or predicted losses.

[0070] The non-anomalous trajectory data may be labeled.

[0071] The context data may further include an internal map and / or an external map.

[0072] The non-anomalous trajectory data may include trajectory data associated with no violations between the agent and the autonomous vehicle.

[0073] The anomalous trajectory data may include trajectory data associated with a violation between the agent and the autonomous vehicle, or trajectory data not associated with no violation between the agent and the self-vehicle.

[0074] The violation may include an event selected from the group including a collision occurring between the agent and the autonomous vehicle, a distance between the agent and the autonomous vehicle being less than a minimum distance threshold, a deceleration of the autonomous vehicle being greater than a deceleration threshold, an acceleration of the autonomous vehicle being greater than an acceleration threshold, and a jerk of the autonomous vehicle being greater than a jerk threshold.

[0075] A transitory or non-transitory computer readable medium comprises instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of the preceding claims. DETAILED DESCRIPTION OF THE INVENTION

[0076] According to the present invention, there is provided a method as set out in the accompanying claims. Also provided is a computer program, a computer and a non-transitory computer-readable storage medium. Further features of the invention will become apparent from the dependent claims and the following description.

[0077] method According to a first aspect, a computer-implemented method for generating a trajectory of an actor comprises: simulating a first scenario including an environment having an ego-vehicle, a set of actors including a first actor, and optionally a set of objects including a first object, using a first trajectory of the first actor; Observing, by a first adversarial reinforcement learning agent, a first observation of an environment (e.g., an ego-vehicle, a second actor in the set, and / or a first object in the set) in response to a first trajectory of a first actor; generating, by the first agent, a second trajectory for the first actor based on the first observation of the observed environment.

[0078] In this manner, a second trajectory is generated by the first agent based on observations of the environment, e.g., the ego-vehicle, a set of actors including or excluding the first actor, and optionally a set of objects including the first object. The second trajectory of the first actor, e.g., for use in another scenario, is an informed, rather than random or systematic, perturbation or modification of the first trajectory, e.g., a maximally informed adversarial perturbation. In this manner, the method generates trajectories more efficiently, and because the generation is informed, the trajectories explore the environment more efficiently, thereby improving the detection of defects in the ego-vehicle and corresponding vehicle control software. For example, trajectories may be generated by learning, heuristics, or extracted from driving statistics and / or their complements. For example, as described in more detail below, trajectories may be generated by rejection sampling, thereby sampling trajectories outside of normal or expected scenarios (i.e., normal space or the complement of (1-N)). In this manner, a scenario may be recreated with usefully generated, e.g., modified, trajectories. Improving the detection of defects in the control software of the ego-vehicle and corresponding vehicles improves the safety of the control software, thereby improving the safety of the corresponding vehicles and / or their occupants. In contrast, conventional trajectory generation methods randomly or systematically search the environment, thereby extending runtime and / or requiring increased computational resources, while potentially failing to detect defects.

[0079] In one example, generating a second trajectory for the first actor by the first agent based on the first observation of the observed environment includes exploring by the first agent outside of a normal space (i.e., a normal or expected scenario), for example, as described below for points E, I, and F.

[0080] In other words, rather than identifying initial scenarios through road testing, the method is used to generate low-probability events, thereby significantly reducing the miles of driving required for validation, verification, etc. Similarly, rather than randomly perturbing the trajectories of actors in scenarios, the method generates these trajectories from a trained adversarial model, which can, for example, simulatedly interact with the environment and react to the AV's actions. In this way, the volume of difficult, low-probability scenarios generated per simulated mile driven and per unit time is increased.

[0081] Thus, the trained adversarial agent generates dynamic actor (e.g., vehicle / pedestrian / cyclist) trajectories, which are challenging for AVs. The adversarial agent learns by interacting with the (simulated) driving environment and the target AV system. Thus, over time, the adversarial agent learns potential weaknesses of the AV and efficiently generates low-probability driving scenarios in which the AV is likely to perform suboptimally. These scenarios may be used to test problems in the target AV system for validation and confirmation purposes, and as training data to further improve the performance of the AV system. Similarly, the method may be used for regression and / or progression testing. Similarly, the method may be used to parameterize deterministic tests.

[0082] The method is a computer-implemented method, i.e. the method is performed by a computer comprising a processor and a memory. Suitable computers are known.

[0083] The method includes simulating a first scenario. Computer-implemented methods (i.e., in silico) that include simulating a scenario are known. Generally, a scenario is a description of a driving situation, including relevant actors, an environment, a goal, and a sequence of events. For example, a scenario may be represented in a 2D bird's-eye view. 1. A scene or environment (e.g., roads, lanes, obstacles); 2. The vehicle and its trajectory; 3. Actors (pedestrians, other vehicles, etc.) and their individual trajectories, 4, optionally consisting of short sequences (a few seconds to tens of seconds) containing four main elements with objects in the scene (traffic lights, stopped bicycles, cars).

[0084] Additional context elements (actors, objects) may be added to better represent the composition of scenes and scenarios.

[0085] The scenario includes an environment having an ego-vehicle, a collection of actors including a first actor (i.e., at least one actor), and optionally a collection of objects including a first object. As will be understood by those skilled in the art, the environment, also known as a scene, typically includes one or more roads with one or more lanes and, optionally, one or more obstacles. Generally, the ego-vehicle is a connected vehicle and / or an automated vehicle, whose behavior is primarily important in test, trial, or operational scenarios. It should be understood that the behavior of the ego-vehicle is defined by its control software (also known as an AV stack). In one example, the first actor is a road user, such as a vehicle, a pedestrian, or a cyclist. Other road users are also known. In one example, the first object includes and / or is infrastructure such as a traffic light or a stopped road user. In one example, the set of actors includes A actors, where A is a natural number greater than or equal to 1, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more. In one example, the set of objects includes O objects, where O is a natural number greater than or equal to 1, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more.

[0086] Simulating the first scenario includes using a first trajectory of a first actor, where it is understood that the actor has an associated trajectory, and the first trajectory may be described by a descriptor, as described below.

[0087] The method includes observing, with a first adversarial reinforcement learning agent (also known herein as an agent or an adversarial agent), a first observation of an environment (e.g., an ego-vehicle, a second actor in the collection, and / or a first object in the collection) in response to a first trajectory of the first actor. That is, the first trajectory of the first actor may cause a change in the environment. For example, the trajectory of the ego-vehicle and / or the trajectory of the second actor may change in response to the first trajectory of the first actor, e.g., to avoid a collision with the first actor. In one example, the first observation of the environment is an observation of the ego-vehicle. In one example, observing the first observation of the environment with the agent includes observing a first behavior of the environment with the agent, the first behavior comprising the first observation. In one example, the method includes providing one or more reinforcement learning agents, such as adversarial and / or non-adversarial RL agents, and cooperating and / or interacting with the first agent, the collection of actors, and / or the collection of objects.

[0088] The method includes generating, by a first agent, a second trajectory for the first actor based on first observations of the observed environment. That is, the first agent learns from the first trajectory of the first actor and the first observations observed in response to the first trajectory, and generates the second trajectory from this learning. In other words, as discussed above, the generation of the second trajectory is informed by the first observations.

[0089] In particular, the inventors identify the following conventional methods.

[0090] i.e., the system does not consider the similarity of the generated scenarios, as it can continue to exploit previously discovered weaknesses in AV behavior (or those that are very similar to already known weaknesses) without discovering new problems. Discovering diverse adversarial scenarios is key to efficient automated problem discovery.

[0091] ii) They do not consider or evaluate how informative the scenario seeds are for each training episode. During the training process, they do not consider or evaluate how informative the generated scenarios are. Similarly, existing systems lack adequate measures to limit catastrophic forgetting while avoiding mode collapse.

[0092] iii) Adversarial scenario starting conditions are typically generated by selecting actor positions randomly or by copying previously discovered difficult scenarios. By predicting difficult or new starting conditions for the AV stack and using this to generate scenario starting conditions in an informed and automated manner, a wide variety of scenarios can be discovered.

[0093] iv. Generally, we do not attempt to output test parameter settings (e.g., regression, progression) or defect reports as one of our direct results.

[0094] v. Use a single adversarial agent (does not consider multiple adversaries working together to create more complex adversarial scenarios).

[0095] vi, considers AV as a black box and uses high-level metrics such as collisions rather than being able to exploit individual subsystems in the AV stack based on the individual performance metrics of the individual subsystems in the AV stack.

[0096] vii. The focus is on causing a collision by any means, regardless of whether the collision is preventable. If the collision is unavoidable (e.g., if an object appears in front of the AV less than the minimum braking distance, or if a pedestrian strikes a stopped AV), the collision is not caused by the AV and does not necessarily represent a problem with the technology used.

[0097] Therefore, as described herein, the inventors improve upon conventional methods, for example, by:

[0098] a,similarity and diversity of generated scenarios (to maximize coverage) - DB of scenario and trajectory descriptors, scenario and trajectory matcher, anomaly detection by reconstructed scenarios or trajectory loss, scenario and trajectory descriptors b. Informed diversification of adversarial scenario seeds and starting conditions (exploration) c, Predictive reward of policy to prevent catastrophic forgetting / Mixed-policy mix or categorical policy d. Learning to convert normal scenarios into abnormal scenarios e, Dynamic time-warping matching of scenarios and learned matching of scenarios f. Two-stage operation, i.e., coarse-to-fine: First, efficiently reduce the search space using a trained, differentiable black-box replica of the AV stack or one or more of its (sub)components, followed by adversarial fine-tuning with the real AV stack in a simulator. g. Detecting problems and deriving actionable items - "field" bugs or defect / bug reports h. Identifying actionable items from problem detection - Setting parameters for regression and progress testing i) Real-world scenarios are easier to recreate and utilize; trained encoders, generic scenario and trajectory descriptors allow existing real-world scenarios to be converted into latent encodings and sampled in an informed manner around them, as opposed to manually recreating the scenarios.

[0099] A. Trajectory and context coding (descriptors) In one example, the method includes defining the generated second trajectory as a series of descriptors of individual locations, e.g., description-location pairs, where the descriptors include one or more components related to the actor or agent, the ego-vehicle, other actors, and the environment. For example, as described with reference to FIG. 1 , the descriptors may be represented as a series of T*(X+N) for T time steps, with XD position encoding and ND encoding relative to other traffic participants, road configuration, and scene context. Optionally, the descriptors may be represented in a coordinate and context normalized, agent-centric, or world-centric representation.

[0100] In one example, the set of descriptors is heuristic and / or learned, i.e., the descriptors may be heuristic (e.g., different fields dedicated to specific information) or learned (e.g., latent encoding of a scene / scenario).

[0101] In one example, the method includes deriving a set of descriptors from data including physical and / or simulated data of a scenario. That is, the descriptors may be derived from both real-world (i.e., physical) data (see below for details on automatically labeling sequential data) and simulated data. This means that they can be used as both inputs to and outputs from the system, as needed. This greatly improves the compatibility of components, facilitating the storage, comparison, and interoperability of real-world data, simulated data, and outputs from the processes described below.

[0102] B, Data labeled In one example, the method includes labeling the data, such as by applying a recognition model to the data, and deriving a set of descriptors from the data includes deriving the set of descriptors from the labeled data. That is, data for generating descriptors is collected and automatically labeled, such as by applying a recognition model (trained and heuristic) to existing sequential data. The recognition model may include image-level semantic segmentation and object detection, optical flow, etc., laser / LiDAR semantic segmentation and object detection, radar object detection / velocity estimation, large-scale scene understanding, etc. Post-processing, smoothing, etc., can be performed using inertial data and vehicle sensor data, etc. Data may be augmented by applying a process with high recall and good precision.

[0103] In general, labeling data using multiple techniques, such as a combination of recognition models and heuristic methods, optionally with a high-quality HD map, is preferable because it allows for independent exploitation of artifacts from individual techniques and more general intermediate features, whereas end-to-end techniques cannot exploit intermediate features.

[0104] Contrary to common expectations, some noise resulting in poor performance of the applied recognition model can be beneficial when labeling data for adversarial scenarios, allowing the generated scenario to reflect the distribution of recognition defects. That is, having noisy labels can be advantageous in terms of directly modeling real-world recognition. For example, pedestrian dropout in one or more frames can be beneficial for training and / or defect detection.

[0105] For example, the localization output may be combined with a map. For example, a recognition model may be used to label road edges or lane markings on one path or trajectory of a road or its lanes, and the labeling may be automatically applied to label other paths or trajectories of the road or its lanes, or another road or its lanes. It should be understood that the agent requires sufficiently accurate and / or precise positions of the ego-vehicle and actors and the layout of the road.

[0106] In one example, the method includes identifying individual positions of vehicles from the physical data and / or identifying individual positions of an ego-vehicle from the simulation data, and deriving the set of descriptors from the data includes deriving the set of descriptors using the identified individual positions of the vehicles and / or the identified individual positions of the ego-vehicle, i.e., applying localization techniques to determine the location of the ego-vehicle within the scene.

[0107] C. Avoiding mode collapse and ensuring novelty In one example, generating the second trajectory of the first actor by the first agent includes generating the second trajectory of the first actor by the first agent predictively or reactively. That is, the second trajectory may be generated predictively (known before performing an action) or reactively (known after performing an action). Reactive methods are generally less efficient, e.g., classifying a mode collapse after it occurs and discarding the entire scenario or agent. However, reactive methods are easier, identifying utility after the fact and acting on it. In contrast, predictive methods are more difficult but more efficient, minimizing wasted resources and time and speeding problem discovery.

[0108] In one example, the method includes determining a mutual similarity between a candidate trajectory for a first actor generated by a first agent and a reference trajectory, and optionally generating a second trajectory for the first actor by the first agent by modifying the candidate trajectory based on the determined mutual similarity, or eliminating the candidate trajectory based on the determined mutual similarity.

[0109] It should be understood that a candidate trajectory is a candidate for a second trajectory, and a reference trajectory may be the first trajectory, or a stored trajectory that is stored in, for example, a database and selectively accessed, etc. For example, a candidate trajectory may be compared to trajectories contained in the database, and the trajectories may be accessed exhaustively or as a subset based on a classification associated with the scenario.

[0110] One simple approach involves a database of trajectory and context descriptors (with the potential use of databases such as EU, NA, etc. that identify many accidents and causes). A matching process (learned and / or heuristic) can determine the similarity of the descriptors (and thus the similarity of the scenarios) and can be used to make decisions (discard the scenario, adjust the scenario, etc.).

[0111] In one example, the method includes rewarding the first agent according to the mutual dissimilarity of the first trajectory and the second trajectory, such that the first agent is rewarded for generating new trajectories.

[0112] D. Matching In one example, the method includes matching the generated second trajectory with a reference trajectory.

[0113] A set of two or more descriptors, each encoding a specific scenario or trajectory of a dynamic agent, can be matched at multiple scales, levels, and granularities. This allows for matching of:

[0114] Matching scenarios with trajectories that are similar in shape but captured at different time scales or different resolutions / number of time steps, and / or Matching trajectories and scenarios originating from environments or scenes that differ in shape but also in composition (road structure, actor and ego-vehicle positions, position within a lane, transitions from areas reserved for pedestrians to areas reserved for vehicles, etc.), and / or Matching or filtering of trajectories and scenarios based first on subportions of the descriptor length, then on different subportions, etc., generating a hierarchical or tree-like family of relationships between different trajectories and scenarios.

[0115] An example of matching involves initial position matching or filtering using dynamic time warping, followed by one or more stages of matching other parts of the descriptor based on heuristics (such as Euclidean distance), learned methods (e.g., contrastive methods or margins), and / or a custom combination of learned and hard-coded rules.

[0116] In one example, matching the generated second trajectory with the reference trajectory includes matching one or more portions of the generated second trajectory with the reference trajectory.

[0117] E, Reconstruction In one example, the method includes the steps of encoding the generated second trajectory, optionally decoding the encoded second trajectory, calculating a reconstruction quality of the decoded second trajectory, and labeling the generated second trajectory according to the calculated reconstruction quality.

[0118] In one example, the method includes the steps of decoding the encoded trajectory, encoding the decoded trajectory, and calculating a reconstruction quality of the encoded trajectory.

[0119] That is, descriptors may be obtained or encoded by a learned method, allowing for the automatic extraction and description of large amounts of sequential data. This is useful for a number of reasons:

[0120] In some cases, the expressive richness of the descriptor can be extended compared to manually created descriptor fields.

[0121] This allows for automated (often self-supervised) processing of sequential data.

[0122] This allows real-world scenarios of interest to be automatically encoded (and then sampled in an informed manner).

[0123] When used in a variational context, any probability distribution can be used to sample.

[0124] The converged trained model may be used to perform anomaly detection by measuring the reconstruction error of the input. Poor reconstruction indicates an anomaly where the scenario being tested is outside the distribution of training scenarios. Anomalies can be interpreted as novel or adversarial scenarios, among others.

[0125] That is, this allows us to determine whether the input (i.e., the generated trajectory) is from within the normal distribution or outside the normal distribution, i.e., whether the agent is trained using the input.

[0126] Thus, generated trajectories within the normal distribution of actions (i.e., of the first actor) will be seen and correctly encoded / decoded, while generated trajectories from outside the normal distribution of actions will not be correctly encoded. There are two options for using this system:

[0127] 1. Encode the trajectory into a latent space representation, decode the latent space representation into a reconstructed trajectory, and measure the reconstruction error.

[0128] 2. Decode the latent space representation into a decoded trajectory, encode the decoded trajectory into a reconstructed latent space representation, and measure the reconstruction error.

[0129] The second option is preferable because it is self-supervised: the input and output are the only components and do not require labeling.

[0130] F, seeding In one example, the method includes seeding an initial state of a first scenario and initializing the first scenario with the seeded initial state.

[0131] RL agents are generally good at exploitation, e.g., eventually finding defects in the AV stack. However, RL agents are generally not good at exploration, which improves the efficiency of testing, etc.

[0132] The inventors have determined that a first RL agent may be induced to explore by providing maximal notice starting conditions, e.g., by training as described herein and rewarding exploration of new states.

[0133] More specifically, generating trajectories and scenarios is computationally inexpensive, while testing within the SIM is computationally expensive. To reduce the search space, Several methods that can be used to discard scenarios in a reactive manner after testing (using some or all of the methods in C, D, and E above); Some methods that can be used to adjust or discard scenarios when they are being tested in a predictive manner (using some or all of the methods in C, D, and E above); There are several methods and procedures that can be used (see below) to beneficially reduce the number of starting or seed conditions.

[0134] The proposed method for reducing the number of seed conditions is shown in Figure 6. A learned conditional trajectory model is trained to predict trajectories or generate (hallucinate) plausible trajectories using a combination of real-world data and / or simulated data and / or previously generated adversarial trajectories.

[0135] At test time, given a new scene layout (e.g., a previously unencountered road configuration or traffic situation or part of a map), the trained model can be used to sample both plausible starting conditions and plausible future trajectory points, given a set of previous trajectory points.

[0136] This enables large-scale informed sampling of scene configurations, scenario seeds, and starting points. Furthermore, it enables informed exploration during reinforcement learning, balancing utilization to improve coverage and minimize the likelihood of catastrophic forgetting and mode collapse.

[0137] In one example, seeding the initial state of the first scenario includes selecting the initial state from a plurality of initial states, i.e., the initial state is intentionally selected rather than randomly or systematically selected, e.g., to optimize the search.

[0138] G, reward In one example, the method includes rewarding the first agent according to the novelty of the generated second trajectory, e.g., short-term novelty and / or long-term novelty, thus rewarding exploration.

[0139] More specifically, the first agent may be rewarded for the novelty of the states visited, an example being a voxelized grid that encodes additional novelty rewards.

[0140] Intrinsic rewards encourage RL agents to explore new states, leading to more diverse trajectories during training. Rewards can be short-term (e.g., episode) or long-term (across the agent's training runs) rewards, or a combination of both short-term and long-term rewards that are novelty balanced against each other using a scaling factor. An example of a long-term novelty measure can be obtained by random network distillation (RND). RND uses two networks: a randomly initialized and untrained convolutional neural network (the random network), and a predictor convolutional neural network (the predictor network) trained during RL training. The predictor network aims to predict the output of the random network for states seen by the RL network. New states introduce large errors into the predictor network's predictions. (Similar to using an encoder and reconstruction loss, RND is trained on the observations of the RL model rather than a static dataset, so the predictor network's inference error is specific to a given RL training run. However, this adds computational overhead to RL training by adding an additional network to train.)

[0141] In one example, the method includes measuring novelty, such as by using random network distillation (RND).

[0142] H, mode collapse In one example, a method includes assessing a modal collapse of a first agent and adapting the first agent based on a result of the assessment.

[0143] Mode collapse is a major problem in deep learning, and even more so in deep reinforcement learning. In the case of adversarial agents and adversarial scenarios, this typically manifests in models that output adversarial strategies that repeatedly explore the same AV stack flaws or loopholes. This is not only highly inefficient, but can also significantly limit the amount of problems (i.e., coverage) that can be discovered. Certain strategies can help mitigate this problem to some extent (see, among others, C, F, and G). Some strategies mitigate mode collapse but also cause catastrophic forgetting (i.e., previous useful adversarial strategies are "forgotten" in favor of new ones).

[0144] One way to effectively mitigate this is to discretize and classify deep reinforcement learning models based on their behavior and a metric that evaluates mode collapse. Using the same matching and filtering strategy as above, we effectively measure the amount of mode collapse of a model during training, both with respect to previous outputs (i.e., a low-variance detector) and with respect to the outputs of other (e.g., database-stored) models (i.e., a low-global diversity detector). Furthermore, by stopping training when mode collapse occurs and classifying and memorizing these models (memorizing their parameter settings), we can more formally demonstrate coverage for a particular class of problems.

[0145] Similarly, mode collapse metrics can be recorded during the training of a particular agent / model. Training can be stopped when mode collapse occurs, but previous states (parameter settings) of the model corresponding to states where the model exhibited higher variance or diversity, i.e., states where the model achieved "better" scores on one or more mode collapse metrics, may be preserved.

[0146] An example of such a method is shown in FIG.

[0147] During training, clone the agent when it collapses to a single usage mode (according to one or more mode collapse metrics) and save the agent's parameter settings (current or past, desired behavior and mode collapse metric scores) to a database. Restart the search using a new search seed. Alternatively, restart training using a reinitialized agent. Repeat the process to find various adversarial scenarios and train multiple adversarial agents for later testing.

[0148] b) During testing, the stored database of adversarial agents can be used to obtain a diverse set of adversarial scenarios for a given starting seed (agent position, road geometry, etc.). This means that we can expand test coverage by testing the AV stack against a more diverse set of usage modes. There is also the potential for a more formal classification of adversarial scenarios and adversarial agent behaviors.

[0149] Claim - A combination of heuristics and learning. Train an agent to discover adversarial behavior. Monitor the novelty and distance of generated trajectories in the latent or description space. If it determines that novelty or variance begins to decrease, save the current or past parameter settings to a DB along with meta-information to classify the type of trajectory being output, and build a DB of parameter settings. Effect - Terminate policy training or inference, switch to a different policy, new seeds, etc., and reinitialize the agent. Monitor the whole thing again. Diminishing returns - bail out. Also - formally identifiable classes / clusters of these policies - can be used for integration / regression testing. For example, mine only a portion of the AV stack. Classify a set of descriptors. Mining - the search for smooth trajectories - is broadly applicable to other environments. Descriptors are used as a form of exchange between real simulated data, inputs, and outputs - all inputs and outputs are descriptors, and parameter settings are a by-product, but we are interested in the parameter settings (these are the model).

[0150] I,Anomalous Style Conversion In one example, the method includes transforming data including physical data and / or simulation data of the scenario with reference to reference data.

[0151] Given one or more sets of (automatically) labeled non-anomalous trajectory data and one or more sets of (automatically) labeled, learned or generated anomalous trajectory data, a model can be trained to convert the non-anomalous trajectory data into anomalous trajectory data. Advantageously, this training is unpaired and weakly supervised, and does not require labeling of relationships between trajectories.

[0152] One example of such a method uses a Cycle-Consistency Generative Adversarial model to transform non-anomalous data so that the distribution of the non-anomalous data matches the distribution of the anomalous data using adversarial and predicted losses, as shown in Figure 9. In other words, the method transforms the distribution of non-adversarial trajectories to match the distribution of adversarial trajectories.

[0153] It should be understood that anomaly simply means that the distributions of the two sets are different. Set A can be transformed so that its distribution better matches the distribution of set B.

[0154] J,EVALUATING DEFECT REPORTS In one example, the method includes outputting a defect report and, optionally, performing an action in response to the output defect report.

[0155] While the overall goal of the system is problem detection, an important part is represented by the actionable items derived from the system's results, especially the failures that occur. Examples of reporting include "field" bugs or bug reports / defect reports, and parameter settings for regression and progression testing (e.g., deterministic scenarios, fixed simulation scenarios).

[0156] Examples of simulation failures that may trigger a report include: Simple examples include collisions, sudden braking, approaching other actors, and lane violations. ● "Abnormal behavior" for descriptors ●Damaged ST restriction and Poor performance of any metric (e.g., prediction metrics / tracking metrics, specific AV component metrics).

[0157] In one example, the fault report includes one or more faults in the control software of the host vehicle, i.e., the corresponding AV.

[0158] K. Reproducing the target scenario See E above.

[0159] In one example, simulating the first scenario includes simulating a target scenario.

[0160] In this way, the target scenario is used as a seed to simulate a new environment (e.g., an airport or a specific city / intersection / time / traffic / object / actor shuttle).

[0161] L, proxy In one example, the method includes approximating the ego-vehicle or a component thereof as a proxy, and simulating the first scenario includes simulating the first scenario using the proxy. In this manner, the ego-vehicle or a component thereof is approximated (down-sampled) to accelerate exploration of a relatively reduced search space to discover broad categories at a lower computational cost before exploring broad categories using the first agent.

[0162] More specifically, the method involves a two-stage coarse-fine operation that first efficiently reduces the search space using a learned, differentiable black-box proxy of the AV stack or one or more of its (sub)components, followed by adversarial fine-tuning with the real AV stack in a simulator.

[0163] Performing actions and observing conditions in a simulated environment (even though it is much cheaper than driving in the real world) can still be expensive and / or time-consuming. This can be due to a) the slow speed of the simulator environment, b) the AV stack operating at a fixed frequency, or c) both.

[0164] Learned proxies for the AV software stack or one or more subcomponents of the AV stack can be used to speed up operation. Two modes of operation are proposed:

[0165] 1. To address limitations arising from the original AV stack, replace the entire AV stack or one or more of its subcomponents with a learned proxy in a simulated environment (Figure 1, bottom).

[0166] 2. If action labels exist or can be obtained, differentiable learned proxies for subcomponents of the AV stack can be used to train adversarial agents with strong direct supervision (Figure 2, bottom), which addresses both types of limitations.

[0167] This is the "coarse" part of the coarse-fine approach because we subsample the search space in an approximate way, using (imperfect) proxies: proxies are only approximators of the distribution of the actual AV stack (or subcomponents) behavior.

[0168] The "fine" part is then represented by fine-tuning an adversarial agent using the original AV stack within the subsampled search space.

[0169] ●2-stage operation (coarse-fine): ■First, subset the problem space by running it using an approximation of the real component (e.g., a trained version of a tracker with less time constraints). ■Then, explore the points in the subset using the full simulation environment.

[0170] Using strong direct supervision, we can target specific categories of actions (again using our earlier convenient trajectory and scenario descriptors). For example, if we want to train an adversarial agent to induce a specific yaw from the planner, we can do this by first training a learned proxy of the planner, freezing the proxy's parameters, and then training the adversarial agent to get the planner proxy to output plans that lead to trajectories that closely match a particular "type" or descriptor.

[0171] M, autoregression In one example, the method comprises: simulating a second scenario using a second trajectory; Observing, by the first agent, a second observation of the environment in response to a second trajectory of the first actor; Optionally, generating, by the first agent, a third trajectory for the first actor based on a second observation of the observed environment.

[0172] In one example, the method includes generating a first trajectory for a first actor by a first agent.

[0173] That is, the method includes repeating the steps of simulating a scenario using the generated trajectory, observing the environment, and generating a trajectory, such that the output of the method becomes the input of the method. In this way, the first agent is trained.

[0174] In one example, the method includes and / or is a method of training an agent, where training the agent includes establishing, by the agent, a relationship between a first trajectory and a first observation.

[0175] N, non-recoverable event In one example, the method includes a step of rewarding the first agent if a second observation of the environment in response to the second trajectory of the first actor rules out an unrecoverable event, such as an unavoidable collision between the ego-vehicle and the first actor (i.e., the ego-vehicle cannot prevent the collision due to physical constraints or laws of physics, etc.).

[0176] Existing solutions focus on causing a collision by any means, without considering whether the collision is preventable. If the collision is unavoidable or unavoidable (e.g., when an object appears in front of the AV less than the minimum braking distance, or when a pedestrian strikes a stopped AV), the collision is not caused by the AV and does not necessarily represent a problem with the technology being used.

[0177] In one example, the method includes the step of cooperating with a second agent by the first agent and / or the step of interacting with a hostile agent or a non-hostile agent by the first agent.

[0178] That is, a first agent may interact with the actions of a second agent and / or objects, i.e., the environment (non-hostile objects / agents).

[0179] Scenario simulation According to a second aspect, there is provided a computer-implemented method for simulating a scenario, the method comprising: The method includes generating a first trajectory of a first actor of a set of actors according to a first aspect; simulating a first scenario including an environment having an ego-vehicle, a set of actors including the first actor, and optionally a set of objects including a first object, wherein simulating the first scenario includes using the generated first trajectory of the first actor; and identifying defects in the ego-vehicle in the first scenario.

[0180] In one example, the method is a method for testing an ego-vehicle, such as installation, assurance, validation, verification, regression and / or progression testing of its control software.

[0181] Vehicle development According to a third aspect, a computer-implemented method for developing an own vehicle includes: simulating a scenario according to the second aspect; and repairing the identified defects of the host vehicle.

[0182] In one example, repairing the identified defect in the host vehicle includes repairing control software of the host vehicle.

[0183] Computer, computer program, and non-transitory computer-readable storage medium A computer according to a fourth aspect includes a processor and memory configured to carry out the methods according to the first, second and / or third aspects.

[0184] A computer program according to a fifth aspect comprises instructions that, when executed by a computer including a processor and a memory, cause the computer to perform the methods according to the first aspect, the second aspect and / or the third aspect.

[0185] A non-transitory computer-readable storage medium according to a sixth aspect includes instructions that, when executed by a computer including a processor and a memory, cause the computer to perform the methods according to the first aspect, the second aspect, and / or the third aspect.

[0186] definition Throughout this specification, the terms "comprising" or "comprises" mean the inclusion of the specified component, but do not exclude the presence of other components. The terms "consisting essentially of" or "consists essentially of" mean the inclusion of the specified component, but exclude other components except for materials present as impurities and unavoidable materials present as a result of the process used to provide the component, and components added for purposes other than achieving the technical effect of the present invention, such as colorants.

[0187] The terms "consisting essentially of" or "consists essentially of" mean the inclusion of certain components but the exclusion of other components.

[0188] Wherever appropriate, depending on the context, use of the words "comprise" or "comprising" may be interpreted to include the meaning of "consisting essentially of" or "consisting essentially of" or "consisting of" or "consisting of."

[0189] The optional features described herein can be used individually or in combination with one another as appropriate, particularly in the combinations set forth in the appended claims. Optional features of each aspect or exemplary embodiment of the present invention described herein are also applicable to all other aspects or exemplary embodiments of the present invention as appropriate. In other words, those skilled in the art who read this specification will recognize that the optional features of each aspect or exemplary embodiment of the present invention are interchangeable and combinable between different aspects and exemplary embodiments. [Brief explanation of the drawings]

[0190] For a better understanding of the present invention and to show how exemplary embodiments thereof may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0191] [Figure 1] 1 shows a schematic illustration of the ego-vehicle scenario. [Figure 2] 2 illustrates schematically the labeling of data captured in the scenario of FIG. 1; [Figure 3] 2 illustrates a schematic diagram of a method for generating new descriptors from the scenario of FIG. 1, according to one or more embodiments. [Figure 4] 4 shows schematically a matcher for use in the method shown generally in FIG. [Figure 5] 1 illustrates a schematic diagram of a method for labeling trajectory data as an anomaly, according to one or more embodiments. [Figure 6]10A and 10B show schematic diagrams of training and testing methods for fixed or recursive trajectory models, respectively. [Figure 7] 1 shows a schematic diagram of a method for random network distillation. [Figure 8] 2 illustrates a schematic diagram of a method or method for training a policy for an agent from the scenario of FIG. 1 using reinforcement learning, according to one or more embodiments. [Figure 9] 1A-1D illustrate schematic diagrams of a method for training and performing an anomaly transformation using a fixed or recursive trajectory model, respectively, according to one or more embodiments. [Figure 10] 1 illustrates a schematic diagram of a method for training an anomaly transformation using a first fixed or recursive trajectory model and a second fixed or recursive trajectory model, according to one or more embodiments. [Figure 11] 1 illustrates a schematic diagram of a method for generating defect reports from reinforcement learning episodes when training an agent, according to one or more embodiments. [Figure 12] 1 illustrates a schematic diagram of a method for generating clusters of descriptors for reinforcement learning episodes when training an agent, according to one or more embodiments. [Figure 13] 1 illustrates a schematic diagram of a method for generating clusters of descriptors for reinforcement learning episodes when training an agent, according to one or more embodiments. [Figure 14] 14 illustrates a schematic diagram of a method for generating new descriptors for a descriptor space that includes the clusters of descriptors of FIGS. 12 and 13, according to one or more embodiments. [Figure 15] 1 illustrates a schematic diagram of a method for moving from multiple clusters to generate new descriptors, according to one or more embodiments. [Figure 16] 1 illustrates a schematic diagram of a method for recreating a scenario, according to one or more embodiments. [Figure 17] 1 illustrates a schematic diagram of a method for training an agent using reinforcement learning in an environment that includes a proxy for an autonomous vehicle software stack, according to one or more embodiments. [Figure 18]1 illustrates a schematic diagram of a method for training an agent using reinforcement learning in an environment that includes proxies for software stack components of an autonomous vehicle, according to one or more embodiments. [Figure 19] The above method is outlined in more detail below. [Figure 20] The above method is outlined in more detail below. [Figure 21] The above method is outlined in more detail below. [Figure 22] The above method is outlined in more detail below.

[0192] Detailed Description of the Drawings 1-22 illustrate a method according to an exemplary embodiment, the method being a computer-implemented method for generating a trajectory for an actor, simulating a first scenario including an environment having an ego-vehicle, a set of actors including a first actor, and optionally a set of objects including a first object, using a first trajectory of the first actor; Observing, by a first adversarial reinforcement learning agent, a first observation of an environment (e.g., an ego-vehicle, a second actor in the set, and / or a first object in the set) in response to a first trajectory of a first actor; generating, by the first agent, a second trajectory for the first actor based on the first observation of the observed environment.

[0193] A. Trajectory and context coding (descriptors) FIG. 1 illustrates a more detailed schematic of a method according to an exemplary embodiment. More specifically, FIG. 1 illustrates a schematic of a scenario encountered by an autonomous vehicle 10, which may be an ego-vehicle 10. The scenario includes one or more actors, and in this particular scenario, there are two actors. The two actors include another vehicle 12 and a pedestrian 14. The pedestrian has a trajectory T, e.g., an agent trajectory that is substantially perpendicular to the sidewalk 16 and moves from the sidewalk 16 to the road 18 along which the ego-vehicle 10 is traveling. In this manner, the agent trajectory intersects with the trajectory of the ego-vehicle. The agent trajectory T is captured as a descriptor 20.

[0194] In other words, in this example, the method includes defining the generated second trajectory as a set of descriptors of individual positions, e.g., description-position pairs, where the description includes one or more components related to the actor or agent, the ego-vehicle, other actors, and the environment. For example, as described with reference to FIG. 1 , the descriptors may be represented as a set of T*(X+N) for T time steps, with XD position encoding and ND encoding relative to other traffic participants, road configuration, and scene context. Optionally, the descriptors may be represented in a coordinate and context normalized, agent-centric, or world-centric representation. In this example, the set of descriptors is heuristic and / or learned. In this example, the method includes deriving the set of descriptors from data including physical data and / or simulation data of the scenario.

[0195] The host vehicle 10 may include multiple sensors 22 and an on-board computer 24. The sensors may include sensors of various modalities, such as radar sensors, image sensors, LiDAR sensors, inertial measurement units (IMUs), and odometry. The computer 24 may include one or more processors and storage devices. The host vehicle may include one or more actuators, such as an engine (not shown), that move the host vehicle along a trajectory.

[0196] B. Data labeling FIG. 2 shows the method of FIG. 1 in more detail and in a simplified diagram.

[0197] In this example, the method includes labeling the data, such as by applying a recognition model to the data, and deriving a set of descriptors from the data includes deriving the set of descriptors from the labeled data, i.e., data for generating descriptors is collected and automatically labeled, such as by applying a recognition model (trained and heuristic) to existing sequential data.

[0198] In this example, the method includes identifying individual positions of vehicles from the physical data and / or identifying individual positions of an ego-vehicle from the simulation data, and deriving the set of descriptors from the data includes deriving the set of descriptors using the identified individual positions of the vehicles and / or the identified individual position of the ego-vehicle, i.e., localization techniques can be applied to determine the position of the ego-vehicle within the scene.

[0199] In other words, unlabeled sequential data 26 is captured by one or more sensors 22 (FIG. 1). The unlabeled sequential data 26 may include image data 26_1, LiDAR data 26_2, radar data 26_3, location information 26_4, and vehicle data 26_5. Optional data 28 may also be provided. The optional data 28 may include an internal map 28_1, an external map 28_2, and field annotations 28_3. The data 26, 28 may be automatically labeled at 30. The result of the automatic labeling is labeled trajectory data 32.

[0200] C. Avoiding mode collapse and ensuring novelty FIG. 3 shows the method of FIG. 1 in more detail and in schematic form.

[0201] In this example, generating the second trajectory of the first actor by the first agent includes predictively or reactively generating the second trajectory of the first actor by the first agent.

[0202] In this example, the method includes determining a mutual similarity between a candidate trajectory for a first actor generated by a first agent and a reference trajectory, and optionally generating a second trajectory for the first actor by the first agent by modifying the candidate trajectory based on the determined mutual similarity, or eliminating the candidate trajectory based on the determined mutual similarity.

[0203] It should be understood that a candidate trajectory is a candidate for a second trajectory, and a reference trajectory may be the first trajectory, or a stored trajectory that is stored in, for example, a database and selectively accessed, etc. For example, a candidate trajectory may be compared to trajectories contained in the database, and the trajectories may be accessed exhaustively or as a subset based on a classification associated with the scenario.

[0204] In this example, the method includes rewarding the first agent according to the mutual dissimilarity of the first trajectory and the second trajectory, such that the first agent is rewarded for generating new trajectories.

[0205] In other words, a descriptor 20 may be generated for each point in a scenario. A point in a scenario may be a time point or a position point of the ego-vehicle. Each point may include the position and pose of each actor or agent, the position and pose of the ego-vehicle 10, and context information. The context information may include an internal map and an external map. A scenario may have multiple points. Therefore, there may be multiple descriptors, and each descriptor may be generated for a point. A trajectory T may be a series of positions and poses of an agent in a scenario.

[0206] Each descriptor 20 may be input to a matcher 34, which is described in more detail with reference to Figure 4 below. At 35, matcher 34 compares the set of descriptors 20 to a descriptor sequence database 36 and determines the degree of similarity (e.g., distance) between the compared sequences. If the agent trajectory sequence is not similar to any in database 36, then at 38 the sequence is stored in database 36. If the agent trajectory sequence is similar, then at 40 the agent trajectory sequence is adjusted or discarded.

[0207] D. Matching FIG. 4 illustrates the method of FIG. 1 in more detail.

[0208] In this example, the method includes matching the generated second trajectory with the reference trajectory.

[0209] An example of matching involves initial position matching or filtering using dynamic time warping, followed by one or more stages of matching other parts of the descriptor based on heuristics (such as Euclidean distance), learned methods (e.g., contrastive methods or margins), and / or a custom combination of learned and hard-coded rules.

[0210] In this example, matching the generated second trajectory with the reference trajectory includes matching one or more portions of the generated second trajectory with the reference trajectory.

[0211] In other words, Figure 4 schematically illustrates the matcher 34 of Figure 3. The matcher 34 may be configured to compare the similarity of two trajectories, for example, trajectory 1 (agent trajectory T) and trajectory 2 (a trajectory stored in database 36). The matcher may include one or more constituent matchers. The constituent matchers may include one or more of a dynamic time warping (DTW) matcher 42_1, a Euclidean distance matcher 42_2, a learned distance matcher 42_3 (which may be a neural network trained to calculate the distance between two columns of points), a custom matcher 42_4 (which may be any other combination of matchers), and a context matcher 42_5.

[0212] E, Reconstruction FIG. 5 shows the method of FIG. 1 in more detail and in schematic form.

[0213] In this example, the method includes the steps of encoding the generated second trajectory, optionally decoding the encoded second trajectory, calculating a reconstruction quality of the decoded second trajectory, and labeling the generated second trajectory according to the calculated reconstruction quality.

[0214] In this example, the method includes the steps of decoding the encoded trajectory, encoding the decoded trajectory, and calculating the reconstruction quality of the encoded trajectory.

[0215] 5 illustrates the training and testing of an autoencoder, more specifically a variational autoencoder (VAE). The VAE may include an encoder 44 and a decoder 46.

[0216] During training, the encoder may be configured to generate descriptors 20 from labeled trajectory data 48. The decoder may be configured to use the descriptors 20 to reconstruct trajectory data 50. The encoder and decoder are trained to reduce or minimize the loss between the reconstructed trajectory data 50 and the labeled trajectory data 48.

[0217] During testing, the reconstructed trajectories are compared to the original labeled trajectories 48, and the reconstruction quality 51 is calculated. If the reconstruction quality is low, e.g., below a threshold, at 52, the data is labeled as an anomaly at 54. Anomalies 54 may be detected because the reconstructed trajectories are outside the trained distribution. Therefore, such anomalies may be good candidates for use in a simulator to test the AV stack.

[0218] F, seeding FIG. 6 shows a more detailed schematic of the method of FIG.

[0219] In this example, the method includes the steps of seeding an initial state of a first scenario and initializing the first scenario with the seeded initial state.

[0220] The proposed method for reducing the number of seed conditions is shown in Figure 6. The learned conditional trajectory model is trained to predict trajectories or generate plausible trajectories (hallucinations) using a combination of real-world data and / or simulation data and / or previously generated adversarial trajectories.

[0221] At test time, given a new scene layout (e.g., a previously unencountered road configuration or traffic situation or part of a map), the trained model can be used to sample both plausible starting conditions and plausible future trajectory points, given a set of previous trajectory points.

[0222] This enables large-scale informed sampling of scene configurations, scenario seeds, and starting points. Furthermore, it enables informed exploration during reinforcement learning, balancing utilization to improve coverage and minimize the likelihood of catastrophic forgetting and mode collapse.

[0223] In this example, seeding the initial state of the first scenario includes selecting the initial state from a plurality of initial states, i.e., the initial state is intentionally selected rather than randomly or systematically selected, e.g., to optimize the search.

[0224] In other words, the method schematically illustrated in Figure 6 is proposed to reduce the number of seed conditions required to generate possible anomalous trajectories.

[0225] A fixed or recursive trajectory model 60 may be trained in a training phase by inputting context data 62, which may include an internal map 63 and an external map 64. Optionally, trajectory seeds 66 may be input using labeled trajectory data 48, and noise 68 may be input using a noise generator 70. A predicted trajectory 72 may be generated, and a prediction or reconstruction loss may be generated. The trajectory model 60 may include a neural network. Parameter settings of the trajectory model 60 may be optimized by minimizing the prediction or reconstruction loss.

[0226] During testing, trajectory model 60 may use context data 62 , noise 68 and trajectory seeds 66 as inputs to generate new trajectory data 74 .

[0227] G, reward FIG. 7 shows the method of FIG. 1 in more detail and in schematic form.

[0228] In this example, the method includes rewarding the first agent depending on the novelty of the generated second trajectory, e.g., short-term novelty and / or long-term novelty, thus rewarding exploration.

[0229] More specifically, the first agent may be rewarded for the novelty of the states visited, an example being a voxelized grid that encodes additional novelty rewards.

[0230] Intrinsic rewards encourage RL agents to explore new states, leading to more diverse trajectories during training. The reward can be a short-term (e.g., episode) reward or a long-term (across the agent's training runs) reward or a combination of both short-term and long-term rewards that are novelty balanced against each other using a scaling factor. An example of a long-term novelty measure can be obtained by random network distillation (RND). RND uses two networks: a randomly initialized and untrained convolutional neural network (the random network), and a predictor convolutional neural network (the predictor network) trained during RL training. The predictor network aims to predict the output of the random network for states seen by the RL network. New states introduce large errors into the predictor network's predictions. (Similar to using an encoder and reconstruction loss, RND is trained on the observations of the RL model rather than a static dataset, so the predictor network's inference error is specific to a given RL training run. However, this adds computational overhead to RL training by adding an additional network to train.)

[0231] In this example, the method includes measuring novelty, such as by using random network distillation (RND).

[0232] H, mode collapse FIG. 8 illustrates the method of FIG. 1 in more detail.

[0233] In this example, the method includes assessing a modal collapse of a first agent and adapting the first agent based on a result of the assessment.

[0234] An example of such a method is shown in FIG.

[0235] a. During training, clone the agent when it collapses to a single usage mode (according to one or more mode collapse metrics) and save the agent's parameter settings (current or past, desired behavior and mode collapse metric scores) to a database. Restart the search using a new search seed. Alternatively, restart training using a reinitialized agent. Repeatedly iterate to find various adversarial scenarios and train multiple adversarial agents for later testing.

[0236] b) During testing, the stored database of adversarial agents can be used to obtain a diverse set of adversarial scenarios for a given starting seed (agent position, road geometry, etc.). This means that we can expand test coverage by testing the AV stack against a more diverse set of usage modes. There is also the potential for a more formal classification of adversarial scenarios and adversarial agent behaviors.

[0237] In other words, Figure 8 schematically illustrates an adversarial agent 76 that can convert states into actions. Each actor in a scenario may be associated with only one agent. For example, each agent may govern the actor's movement depending on a given state. An action may be a future position to which the actor has moved, or the actor's velocity, pose, etc. The agent 76 may include a machine learning algorithm, which may be a neural network.

[0238] The AV software stack 78 may include modules including perception and control. The AV software stack may be provided to the computer 24 (FIG. 1) at runtime. The AV software stack 78 may be configured to observe and perceive an environment including an actor controlled by the agent 76 and control the ego-vehicle 10 according to the agent trajectory. In other words, the agent 76 generates the actor's trajectory according to changes in conditions including the AV (ego-vehicle).

[0239] The agent 76 may be trained using reinforcement learning or deep reinforcement learning within an environment including an AV software stack 78. Contextual data may also be provided within the environment. For example, there may not be a target state that the agent is trained to match depending on previous input states. Instead, a reward may be used when an episode (e.g., a sequence of states and actions) achieves a goal. For example, the goal may include an adversarial goal, such as an actor colliding with the ego-vehicle. This may occur if an episode includes an actor (e.g., a pedestrian) suddenly jumping off the sidewalk into the road and the ego-vehicle's trajectory. In this manner, an adversarial event may occur. If the AV stack has a flaw that causes the ego-vehicle not to change course to avoid the actor, this may be captured as an adversarial event.

[0240] Other adversarial events may also occur, including events selected from the group including: a collision between the agent (or actor) and the autonomous vehicle; the distance between the agent and the autonomous vehicle being less than a minimum distance threshold; the deceleration of the autonomous vehicle being greater than a deceleration threshold; the acceleration of the autonomous vehicle being greater than an acceleration threshold; and the jerk of the autonomous vehicle being greater than a jerk threshold.

[0241] Each episode may end with an adversarial event or a failure of the AV software stack.

[0242] Observations may be made and descriptors of the actor's states and actions may be generated at 80. The descriptors may be generated by an encoder. A matcher, which may include the matcher of FIG. 3, may compare the descriptors to descriptors from the descriptor sequence database 36. The descriptor sequence database 36 may include multiple descriptors, each of which includes a descriptor from a previous episode.

[0243] New episodes can be compared by reinitializing the agent and re-running the reinforcement learning loop, which generates a new episode and multiple new descriptors.

[0244] At 82, it is determined whether there is mode collapse. Mode collapse may be determined when there is low variance between the compared episodes. Low variance may be classified as variance that is below a variance threshold or convergent variance.

[0245] If there is no mode collapse, e.g., if the agent generates a new adversarial episode, training continues. If there is a mode collapse, e.g., if the adversarial episode matches a previous adversarial episode, the agent is cloned at 84. At 86, the parameter settings (e.g., combinations of weights in the network) of the agent that caused the adversarial event may be stored in a parameter database. At 88, a new search strategy or trajectory may be sampled for the cloned agent. The new search strategy may be seeded from an initial state derived from descriptors from the descriptor sequence database 36.

[0246] It is important to note that mode collapse is typically seen as a negative. However, mode collapse in this scenario is used to identify anomalous adversarial events, which can then be used to improve the AV stack using a simulator. In this way, the replicated adversarial agent can be used in the simulator to improve the AV software stack.

[0247] I,Anomalous Style Conversion FIG. 9 illustrates the method of FIG. 1 in more detail.

[0248] In this example, the method includes transforming data including physical data and / or simulation data of a scenario with reference to reference data.

[0249] One example of such a method uses a cycle-consistent generative adversarial model to transform non-anomalous data so that the distribution of the non-anomalous data matches the distribution of the anomalous data using adversarial and prediction losses, as shown in Figure 9. In other words, the method transforms the distribution of non-adversarial trajectories to match the distribution of adversarial trajectories.

[0250] It should be understood that anomaly simply means that the distributions of the two sets are different. Set A can be transformed so that its distribution better matches the distribution of set B.

[0251] The anomalous trajectory may then be explored within the simulator to determine whether it is associated with the occurrence of an adversarial event, for example, a collision between the agent and an AV or ego vehicle.

[0252] Referring to Figure 10, a method for training an anomaly transformation using a cycle-consistent GAN or cycle-consistent generative adversarial model is provided. This model may use similar features to the method and model of Figure 9, and therefore, redundant description will be omitted for the sake of brevity.

[0253] The models may include a first model 102 (or Model A), also referred to as a fixed or recursive trajectory model A, and a second model 104 (or Model B), also referred to as a fixed or recursive trajectory model B. The first model 102 may be configured to generate predicted anomalous trajectory data 94 that is compared with anomaly-labeled trajectory data 96 to generate an adversarial loss 100. The predicted anomalous trajectory data 94 may be input to a second model 104 configured to generate reconstructed non-anomalous trajectory data 106. The reconstruction loss 108 and the adversarial loss 100 may be obtained by comparing the reconstructed non-anomalous trajectory data with the non-anomalous-labeled trajectory data 92. Parameter settings of the second model may be modified to reduce the reconstruction loss 108 and the adversarial loss 100.

[0254] In this manner, new anomalies or potential adversarial events can be synthesized using cycle GANs, etc. Once synthesized, the new anomalies can be run through the simulator to test whether they are adversarial scenarios that could result in, for example, a failure of the AV stack 10.

[0255] J,EVALUATING DEFECT REPORTS FIG. 11 shows the method of FIG. 1 in more detail and diagrammatically.

[0256] In this example, the method includes outputting a defect report and, optionally, performing an action in response to the output defect report.

[0257] In this example, the fault report includes one or more faults in the control software of the host vehicle, i.e., the corresponding AV.

[0258] In other words, Figure 11 illustrates schematically that faults may be detected at 108 (e.g., by a fault detector) within a reinforcement learning environment. The reinforcement learning environment may be within a simulator. Faults may include collisions, hard braking, following too close to other actors, lane violations, etc. In other words, faults may be adversarial events as described herein.

[0259] At 110, the AV software stack is interrogated as to whether it has failed or malfunctioned. A defect report may be generated at 112. The defect report at 112 may be stored in a defect dataset at 114.

[0260] Referring to Figure 12, a method similar to that shown in Figure 11 is provided. In Figure 12, similar features are not described, but only differences from Figure 11 are described. One such difference is the inclusion of a cluster database 116. The cluster database 116 may include clusters of adversarial events.

[0261] Multiple reinforcement learning episodes, sets of reinforcement learning episodes, or points of reinforcement learning episodes may be clustered together, and the points in the cluster may be added to the cluster database 116.

[0262] Generating new latent adversarial descriptors Figure 13 shows a schematic of how descriptors of adversarial events observed during reinforcement learning of an agent 76 are generated and stored. At 80, states and actions within an episode that result in a violation (also called an adversarial event) are observed and described. Descriptors 20 encoded from actions and states are stored in a cluster database 116. As described above, actions and states are clustered according to their associated episode.

[0263] 14 shows a schematic representation of the cluster database 116, represented as a descriptor space envelope 120. Within the descriptor space envelope 120, clusters C of descriptors 20 are provided. A cluster includes descriptors that are determined to match each other within a matching threshold. The clusters may be determined using a clustering algorithm, which may be an unsupervised clustering algorithm.

[0264] The objective of the subject matter of this disclosure is to explore the descriptor space envelope to obtain more descriptors of potential adversarial scenarios that can be tested in a simulator to learn new faults in the AV software stack. Exploring the descriptor space envelope on an AV can take a very long execution time, making it computationally intensive and expensive.

[0265] Alternatively, in one or more embodiments, the descriptor space envelope 120 may be explored by moving from the currently known cluster C. There are various ways in which this can be achieved.

[0266] One such method involves determining new descriptors: a direction is determined from the centroid of the cluster, and new descriptors are generated for incremental positions moving from the centroid in that direction. This can be understood with reference to Equation A below:

[0267] Formula A: New Descriptor = (C1 + C2 + ... + CN) / N + unit_direction_away_from_super_barycenter × M

[0268] In Equation A, C1 is the first descriptor, C2 is the second descriptor, CN is the Nth descriptor, and N is the total number of descriptors. This part of Equation A efficiently calculates the center of gravity. Also, unit_direction_away_from_super_barycenter is the direction, such as up or down. Furthermore, M is the distance away from the center of gravity.

[0269] Another way to search the descriptor space envelope 120 is to use Equation B:

[0270] Formula B: New descriptor = SDF + unit_direction_away_from_super_barycenter × M

[0271] In Equation B, SDF is a signed distance function. The other parameters are the same as those in Equation A.

[0272] Another way to search the descriptor space envelope 120 is to use Equation C:

[0273] Formula C: New Descriptor

number

number

[0274] Figure 15 shows an extension of the idea of ​​exploring the descriptor space envelope from a single cluster shown in Figure 14. There are three clusters in Figure 15, and the step of moving from a cluster is to find the union (C 1 UC 2 UC 3 ), determining the difference between the cluster space C and the union using formula D, determining the centroid of the difference, and generating a new descriptor as the descriptor at the centroid of the difference.

[0275] Formula D:C\(C 1 UC 2 UC 3 )

[0276] The advantage of this approach is that it reduces the chance of finding another cluster within the descriptor space envelope.

[0277] At a high level, the framework can be described algorithmically as follows:

[0278] Initialize the set of solution clusters C={} For N meta-episodes: Initialize the probabilistic policy P with the convergence temperature α and an empty replay buffer D (for invalidation policies) Initialize the infraction buffer B={} Run the episode until convergence of P (i.e., policy collapse), which is detected via α. ■Until the episode end condition (policy collapse) is met: Observe state s, perform action a, receive reward r, and observe new state s ●Store the transition (s, a, r, s') in D ■ Guide the policy to move away from the solutions barycentres in C (if C is non-empty) ■Optimize policy P based on transitions uniformly sampled from D ■Memorize each episode that leads to a violation in B. Cluster new solutions (e.g., descriptors) of B while cross-checking them with existing solutions C ○When a new solution is discovered, add it to C

[0279] where C is the cluster, N is the number of meta-episodes, P is the agent's policy, α is the convergence temperature or convergence variance, D is the replay buffer, s is the state input to the agent, a is the action output by the agent, r is the reward given to the agent, and s' is the new state generated by the AV software stack (or subcomponent) or proxy (or subcomponent).

[0280] K. Reproducing the target scenario Figure 16 shows in more detail the method of Figure 1. See also Figure 6.

[0281] In this example, simulating the first scenario includes simulating the target scenario.

[0282] Referring to FIG. 13, a method is provided for generating new trajectory data.

[0283] In the method, target scenario context data 118 may include an internal map 63 and an external map 64. The context data 118 may be input to a fixed or recursive trajectory model 119. Optional trajectory seeds 120 may be input to the model 119 from target scenario trajectory data 122. Optional noise 68 may also be input to the model 119 from a noise generator 70. The model 119 may be configured to output new trajectory data 124.

[0284] L, proxy Figures 17 and 18 illustrate in more detail the method of Figure 1. See also Figure 21, in which the AV stack proxy is labelled as Stack-Lite.

[0285] In this example, the method includes approximating the ego-vehicle or a component thereof as a proxy, and simulating the first scenario includes simulating the first scenario using the proxy. In this manner, the ego-vehicle or a component thereof is approximated (down-sampled) to accelerate exploration of a relatively reduced search space to discover broad categories at a lower computational cost before exploring broad categories using the first agent.

[0286] More specifically, the method involves a two-stage coarse-fine operation that first efficiently reduces the search space using a learned, differentiable black-box proxy of the AV stack or one or more of its (sub)components, followed by adversarial fine-tuning with the real AV stack in a simulator.

[0287] Performing actions and observing conditions in a simulated environment (even though it is much cheaper than driving in the real world) can still be expensive and / or time-consuming. This can be due to a) the slow speed of the simulator environment, b) the AV stack operating at a fixed frequency, or c) both.

[0288] Learned proxies for the AV software stack or one or more subcomponents of the AV stack can be used to speed up operation. Two modes of operation are proposed:

[0289] 1. To address limitations arising from the original AV stack, replace the entire AV stack or one or more of its subcomponents with a learned proxy in a simulated environment (Figure 1, bottom).

[0290] 2. If action labels exist or can be obtained, differentiable learned proxies for subcomponents of the AV stack can be used to train adversarial agents with strong direct supervision (Figure 2, bottom), which addresses both types of limitations.

[0291] This is the "coarse" part of the coarse-fine approach because we subsample the search space in an approximate way, using (imperfect) proxies: proxies are only approximators of the distribution of the actual AV stack (or subcomponents) behavior.

[0292] The "fine" part is then represented by fine-tuning an adversarial agent using the original AV stack within the subsampled search space.

[0293] ●2-stage operation (coarse-fine): ■First, subset the problem space by running it using an approximation of the real component (e.g., a trained version of a tracker with less time constraints). ■Then, explore the points in the subset using the full simulation environment.

[0294] Using strong direct supervision, we can target specific categories of actions (again using our earlier convenient trajectory and scenario descriptors). For example, if we want to train an adversarial agent to induce a specific yaw from the planner, we can do this by first training a learned proxy of the planner, freezing the proxy's parameters, and then training the adversarial agent to get the planner proxy to output plans that lead to trajectories that closely match a particular "type" or descriptor.

[0295] In other words, Figure 17 shows four different methods: The first method is the reinforcement learning method of the agent 76 introduced in Figure 8.

[0296] Using the first method, a series of observations 130 observed by the AV software stack 78 and a series of actions 132 performed by the AV stack 130 in response to the observations are generated in the second method.

[0297] In the third method, an AV stack proxy 134 is used instead of the AV software stack 78. The AV stack proxy may be a machine learning model such as a neural network. The neural network may be a convolutional neural network (CNN).

[0298] The AV stack proxy 134 may be trained according to a third method. The AV stack proxy 134 may be trained by generating predicted actions 136 based on input observations 130. A loss 138 between the predicted actions 136 and the actions generated by the second method may be obtained. Parameter settings of the AV stack proxy may be optimized to reduce or minimize the loss 138.

[0299] In the fourth method, reinforcement learning of the agent 76 is performed using states and rewards generated by the AV stack proxy 134 in the simulator.

[0300] Because the AV stack proxy is smaller than the entire AV software stack, anomalies and adversarial scenarios can be determined more quickly. It should be understood that anomalies discovered using the AV stack proxy 134 may be considered approximations. To determine whether a scenario is actually adversarial, a first method may be used to verify the anomaly as an adversarial scenario in which the AV software stack 78 has failed.

[0301] The approximations of the adversarial events may form clusters in the manner shown in Figure 15, and each cluster includes a centroid. Once the clusters are discovered using the coarse approximations by the AV stack proxy 134, the method according to Figure 16 (and Figure 14) may be used to explore the descriptor space and discover new potential adversarial scenarios that can be tested using the full AV software stack 78 on a simulator. This approach is computationally more efficient and reduces the time required to explore the descriptor space.

[0302] The same approach can be used with subcomponents 140 of the AV software stack, for example, semantic segmentation or object recognition.

[0303] Referring to Figure 18, methods are provided for obtaining approximate failures of subcomponents 140 of an AV software stack. Figure 17 shows three methods in schematic form.

[0304] In the first method, observations 130 are input to an AV software stack subcomponent 140, which generates actions 132 in response.

[0305] In a second method, the observations 130 and actions 132 form collected training data. An AV stack subcomponent proxy 142 is trained using the collected training data. Specifically, the AV stack subcomponent proxy 142 uses the observations 130 to generate predicted actions. A loss between the predicted actions 136 and the actions 132 is determined. Parameter settings of the AV stack subcomponent proxy 142 are trained to reduce or minimize the loss 138. The AV stack subcomponent proxy 142 may be or include a machine learning model, such as a neural network. The neural network may be a convolutional neural network (CNN).

[0306] A third method may be a supervised training method with learned subcomponent proxies 142 .

[0307] The learned subcomponent proxy 142 may generate actions based on actions 148 from the agent 76. An action loss 144 and an action classification loss 146 may be calculated to train the agent 76.

[0308] Figure 19 shows in more detail the method of Figure 1. In particular, Figure 18 shows nine scenarios that were simulated using a seed to explore the ego-vehicle's response.

[0309] Figure 20 illustrates in more detail the method of Figure 1. In particular, Figure 20 illustrates a scenario involving multiple candidate trajectories for a first actor (pedestrian). However, because the starting points of the multiple candidate trajectories are the same, the first agent is incentivized to change their starting points while eliminating unavoidable collisions between the ego vehicle and the first actor, such as in front of a truck.

[0310] Figure 21 illustrates in more detail the method of Figure 1. In Figure 21, a stack light may correspond to an AV software stack proxy or an AV software stack subcomponent proxy.

[0311] Figure 22 is a graph of the number of events (trajectories) generated as a function of time according to the method of Figure 1. In particular, the method generates over 300 events in approximately 13 minutes, thereby improving the detection of defects in the host vehicle and the corresponding vehicle's control software.

[0312] While preferred embodiments have been shown and described, it should be understood by those skilled in the art that various changes and modifications can be made therein without departing from the scope of the invention as defined in the appended claims and as set forth above.

[0313] At least some of the exemplary embodiments described herein may be configured, partially or entirely, using dedicated hardware. As used herein, terms such as "component," "module," or "unit" may include, but are not limited to, hardware devices, such as circuits, field programmable gate arrays (FPGAs), or application specific integrated circuits (ASICs), in the form of discrete or integrated components, that perform particular tasks or provide related functionality. In some embodiments, the described elements may be configured to reside on tangible, persistent, addressable storage media and to execute on one or more processors. In some embodiments, these functional elements may include components such as, by way of example, software components, object-oriented software components, class components, task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. While the exemplary embodiments are described with reference to components, modules, and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features are described herein, and the described features may be combined in any suitable combination. In particular, features of any one exemplary embodiment may be combined as appropriate with features of any other embodiment, unless they are mutually inconsistent. Throughout this specification, the terms "comprising" or "comprises" mean the inclusion of the specified component, but do not exclude the presence of other components.

[0314] Attention is directed to all articles and documents related to this application, filed contemporaneously or prior to this application, and open to public inspection herewith, and the contents of all such articles and documents are incorporated herein by reference.

[0315] All features disclosed in this specification (including the accompanying claims, abstract and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, unless at least some of such features and / or steps are mutually inconsistent.

[0316] Each feature disclosed in this specification (including the accompanying claims, abstract, and drawings), unless otherwise stated, may be replaced by alternative features serving the same, equivalent, or similar purpose. Thus, unless otherwise stated, each disclosed feature is only an example of a generic series of equivalent or similar features.

[0317] The invention is not limited to the details of the foregoing embodiments, but extends to any novel feature, or any novel combination of such features, disclosed in this specification (including the accompanying claims, abstract and drawings), or to any novel or novel combination of any method or process steps so disclosed.

[0318] The subject matter of the present disclosure may be expressed by the following:

[0319] 1. A computer-implemented method for generating a trajectory of an actor, the method comprising: simulating a first scenario including an environment having an ego-vehicle, a set of actors including a first actor, and optionally a set of objects including a first object, using a first trajectory of the first actor; Observing, by a first adversarial reinforcement learning agent, a first observation of an environment (e.g., an ego-vehicle, a second actor in the set, and / or a first object in the set) in response to a first trajectory of a first actor; generating, by the first agent, a second trajectory for the first actor based on the first observation of the observed environment.

[0320] 2. A method according to any of the preceding items, comprising defining the generated second trajectories as a set of descriptors for each location, for example as pairs of description and location.

[0321] 3. The method according to item 2, wherein the set of descriptors is heuristic and / or learned.

[0322] 4. The method of any one of items 2-3, comprising deriving a set of descriptors from data including physical data and / or simulation data of a scenario.

[0323] 5. The method of claim 4, further comprising labeling the data, such as by applying a recognition model to the data, and deriving a set of descriptors from the data comprises deriving a set of descriptors from the labeled data.

[0324] 6. The method according to any one of items 4 to 5, comprising identifying individual positions of the vehicle from the physical data and / or identifying individual positions of the ego-vehicle from the simulation data, and deriving the set of descriptors from the data comprises deriving the set of descriptors using the identified individual positions of the vehicle and / or the identified individual position of the ego-vehicle.

[0325] 7. The method of any one of the preceding items, wherein the step of generating the second trajectory of the first actor by the first agent includes the step of predictively or reactively generating the second trajectory of the first actor by the first agent.

[0326] 8. The method of item 7, including a step of determining a mutual similarity between a candidate trajectory of the first actor generated by the first agent and a reference trajectory, and optionally a step of generating a second trajectory of the first actor by the first agent by modifying the candidate trajectory based on the determined mutual similarity, or a step of eliminating the candidate trajectory based on the determined mutual similarity.

[0327] 9. The method according to any one of items 7 to 8, comprising the step of rewarding the first agent according to the mutual dissimilarity between the first trajectory and the second trajectory.

[0328] 10. The method of any one of the preceding items, comprising matching the generated second trajectory with a reference trajectory.

[0329] 11. The method according to item 10, wherein the step of matching the generated second trajectory with the reference trajectory includes the step of matching one or more portions of the generated second trajectory with the reference trajectory.

[0330] 12. The method of any one of the preceding items, comprising the steps of encoding the generated second trajectory, and optionally decoding the encoded second trajectory, calculating a reconstruction quality of the decoded second trajectory, and labeling the generated second trajectory according to the calculated reconstruction quality.

[0331] 13. The method of any one of the preceding items, comprising the steps of decoding the encoded trajectory, encoding the decoded trajectory, and calculating the reconstruction quality of the encoded trajectory.

[0332] 14. The method of any one of the preceding items, comprising the steps of: seeding an initial state of the first scenario; and initializing the first scenario with the seeded initial state.

[0333] 15. The method of claim 14, wherein the step of seeding an initial state of the first scenario includes the step of selecting an initial state from a plurality of initial states.

[0334] 16. The method of any one of the preceding items, comprising a step of rewarding the first agent depending on the novelty of the generated second trajectory, for example short-term novelty and / or long-term novelty.

[0335] 17. The method according to item 16, comprising measuring novelty, such as by using random network distillation (RND).

[0336] 18. The method of any one of the preceding items, comprising the steps of assessing a modal collapse of the first agent and adapting the first agent based on the results of the assessment.

[0337] 19. The method of any one of the preceding items, comprising the step of transforming data including physical data and / or simulation data of the scenario by reference to reference data.

[0338] 20. The method of any one of the preceding items, including the steps of outputting a defect report, and optionally performing an action in response to the output defect report.

[0339] 21. A method according to any one of the preceding items, comprising approximating an ego-vehicle or a component thereof as a proxy, and simulating the first scenario comprises simulating the first scenario using the proxy.

[0340] 22. Generating a first trajectory of a first actor by a first agent; and / or simulating a second scenario using the second trajectory; Observing, by the first agent, a second observation of the environment in response to a second trajectory of the first actor; 10. The method of any one of the preceding claims, optionally comprising generating, by the first agent, a third trajectory for the first actor based on a second observation of the observed environment.

[0341] 23. The method of item 22, further comprising a step of rewarding the first agent if the second observation of the environment according to the second trajectory of the first actor excludes an unrecoverable event, for example, an unavoidable collision between the ego-vehicle and the first actor.

[0342] 24. The method of any one of the preceding items, including the step of cooperating with a second agent by the first agent, and / or the step of interacting with a hostile agent or a non-hostile agent by the first agent.

[0343] 25. A method according to any one of the preceding items, wherein the step of generating by the first agent a second trajectory for the first actor based on a first observation of the observed environment includes a step of exploring by the first agent outside of normal space.

Claims

1. 1. A computer-implemented method for generating new adversarial scenarios and agents, including autonomous vehicles, comprising: performing reinforcement learning to train the agent using a proxy for an autonomous vehicle software stack in a reinforcement learning environment to generate one or more episodes representing adversarial scenarios each terminating in a failure of the proxy for the autonomous vehicle software stack; generating a plurality of descriptors based on the or each episode; clustering a plurality of descriptors for the or each episode; storing the clusters of the plurality of descriptors in a database; generating new descriptors by moving from the clusters of descriptors in descriptor space.

2. The step of moving from the cluster of descriptors in the descriptor space comprises: identifying the centroids of the clusters; moving a unit amount in a unit direction from the centroid to a new descriptor location; generating the new descriptor as a descriptor at the new descriptor location.

3. The step of moving from the cluster of descriptors in the descriptor space comprises: identifying established boundaries of the clusters; moving a unit amount in a unit direction from the boundary to the location of a new descriptor; generating the new descriptor as a descriptor at the new descriptor location.

4. The step of moving from the cluster of descriptors in the descriptor space comprises: identifying established boundaries of the clusters; moving a unit amount from the boundary in a local normal direction to the location of a new descriptor; generating the new descriptor as a descriptor at the new descriptor location.

5. The computer-implemented method of claim 3 , wherein the established boundaries are identified using a signed distance function.

6. The one or more episodes include a plurality of episodes, and clustering the plurality of episodes includes generating a plurality of clusters, storing the clusters includes storing the plurality of clusters in the database, and moving from a cluster includes: determining a union between each cluster; determining the difference between the cluster space and the union; determining the centroid of the differences; 2. The computer-implemented method of claim 1, further comprising: generating the new descriptor as a descriptor at the centroid of the differences; and moving from the plurality of clusters by

7. generating a seed state from the new descriptor; The computer-implemented method of claim 1 , further comprising: re-running reinforcement learning using the seed state, generating the plurality of descriptors, and storing the plurality of descriptors.

8. reinitializing the agent; 10. The computer-implemented method of claim 1, further comprising: re-performing reinforcement learning using a re-initialized agent, generating the plurality of descriptors, and storing the plurality of descriptors.

9. The computer-implemented method of claim 1 , wherein the environment further comprises context data.

10. modifying the context data of an environment; 10. The computer-implemented method of claim 9, further comprising: re-performing reinforcement learning using the modified context data, generating the plurality of descriptors, and storing the plurality of descriptors.

11. The computer-implemented method of claim 1 , wherein the episode includes a plurality of points, each point including a state output by an environment and an action output by the agent.

12. The computer-implemented method of claim 11 , wherein generating the plurality of descriptors comprises encoding each of the plurality of points into a latent space.

13. 2. The computer-implemented method of claim 1, wherein the fault comprises an event selected from the group consisting of: a collision between the agent and the autonomous vehicle software stack occurring; a distance between the agent and the autonomous vehicle software stack being less than a minimum distance threshold; a deceleration of the autonomous vehicle software stack being greater than a deceleration threshold; an acceleration of the autonomous vehicle software stack being greater than an acceleration threshold; and a jerk of the autonomous vehicle software stack being greater than a jerk threshold.

14. The computer-implemented method of claim 1, wherein the proxy comprises a convolutional neural network.

15. 1. A computer-implemented method for generating agents from a scenario including an autonomous vehicle, comprising: providing an agent trained using reinforcement learning in an environment with a proxy for an autonomous vehicle software stack; and performing reinforcement learning to optimize the agent using a fully autonomous vehicle software stack on which the proxy is based.

16. 15. The computer-implemented method of claim 14, wherein providing an agent comprises providing the agent trained upon executing the computer-implemented method of claim 1.

17. 10. A transitory or non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.