Autonomous vehicle planning

Through game tree search algorithm and behavioral model, combined with machine learning and reverse planning, autonomous vehicles can effectively plan manipulation sequences in complex scenarios, solving the problems of insecurity and inefficiency of navigation, and achieving higher navigation safety and efficiency .

CN112888612BActive Publication Date: 2024-11-01FIVE AI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980066757.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-10-16
Filing Date
2019-10-16
Publication Date
2024-11-01
Estimated Expiration
2039-10-16

AI Technical Summary

Technical Problem

It is difficult for autonomous vehicles to effectively plan the manipulation sequence in complex driving scenarios, especially when considering the expected behavior of other vehicles and subjects, resulting in unsafe and inefficient navigation.

Method used

Using a reasoning framework, the game tree search algorithm and behavior model is used to determine the appropriate manipulation sequence by receiving sensor signals, processing driving scenarios, executing tree search algorithms and generating control signals. The framework includes the use of machine learning models to simulate the expected behavior of external subjects and the application of reverse planning methods and target recognition techniques to optimize manipulation decisions.

Benefits of technology

Improve the navigation safety and efficiency of autonomous vehicles in complex scenarios, reduce the risk of conflict with other vehicles through real-time prediction and optimization of manipulation strategies, and improve decision-making capabilities in uncertain environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112888612B_ABST
    Figure CN112888612B_ABST
Patent Text Reader

Abstract

An autonomous vehicle (AV) planning method includes: receiving sensor input related to the AV; processing the AV sensor input to determine an encountered driving scenario; in an AV planner, executing a tree search algorithm to determine an AV maneuver sequence corresponding to a path through a constructed game tree; and generating an AV control signal for executing the determined AV maneuver sequence; wherein the game tree has a plurality of nodes representing expected states of the encountered driving scenario, and an expected driving scenario state of each child node is determined by updating the driving scenario state of a parent node of the child node based on (i) candidate AV maneuvers and (ii) expected behavior of at least one external agent in the encountered driving scenario.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to autonomous vehicle (AV) maneuver planning. Background Art

[0002] An autonomous vehicle, also known as a driverless vehicle, is a vehicle that has a sensor system for monitoring its external environment and a control system capable of using these sensors to automatically make and implement driving decisions. This includes, in particular, the ability to automatically adjust the vehicle's speed and direction of travel based on input from the sensor system. Fully autonomous or "driverless" vehicles have sufficient decision-making capabilities to operate without any input from a human driver. However, the term autonomous vehicle as used herein also applies to semi-autonomous vehicles, which have more limited autonomous driving decision-making capabilities and therefore still require some degree of supervision from a human driver. Summary of the invention

[0003] In order to safely and efficiently navigate encountered driving scenarios, an autonomous vehicle planner needs to be able to plan sequences of maneuvers to be performed in the encountered scenarios. The present invention provides a reasoning framework that allows a computer-implemented AV planner to reason about the possible effects of different sequences of maneuvers in a driving scenario it encounters, taking into account the expected behavior of other vehicles / agents, so that it can determine the appropriate sequence of maneuvers to be performed in the scenario (ego vehicle maneuvers).

[0004] Various aspects of the present invention are directed to an autonomous vehicle (AV) planning method, comprising:

[0005] Receive sensor signals related to AV (input);

[0006] Process sensor inputs to determine encountered driving scenarios;

[0007] In the AV planner, executing a tree search algorithm to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; and

[0008] generating an AV control signal for executing the determined AV manipulation sequence;

[0009] The game tree has multiple nodes representing expected states of encountered driving scenarios, and the expected driving scenario state of each child node is determined by updating the driving scenario state of the child node's parent node based on (i) candidate AV manipulations and (ii) expected behavior of at least one external subject (actor) in the encountered driving scenario.

[0010] An expected behavior of the external agent is simulated by applying a generative behavior model to one or more observed parameters of the external agent derived from the sensor signal.

[0011] The generative behavior model may also be applied to one or more ego vehicle parameters (to model the response of another agent to the ego vehicle implementing a planning method or otherwise being controlled by a planning method) and / or one or more parameters of the driving scenario (such as road layout / other driving environment parameters to model the response of another agent to its environment).

[0012] In a first aspect of the invention, the generative behavior model is a machine learning (ML) model that has been trained based on examples of observed real-world driving behavior.

[0013] That is, a first aspect provides an autonomous driving vehicle (AV) planning method, comprising:

[0014] receiving sensor inputs related to the AV;

[0015] Processing AV sensor inputs to determine encountered driving scenarios;

[0016] In the AV planner, executing a tree search algorithm to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; and

[0017] generating an AV control signal for executing the determined AV manipulation sequence;

[0018] wherein the game tree has a plurality of nodes representing expected states of an encountered driving scenario, and an expected driving scenario state of each child node is determined by updating the driving scenario state of a parent node of the child node based on (i) a candidate AV maneuver and (ii) an expected behavior of at least one external agent in the encountered driving scenario; and

[0019] Therein, the expected behavior of the external subject is simulated by applying a generative behavior model to one or more observed parameters of the external subject derived from sensor input; the generative behavior model is a machine learning (ML) model that has been trained based on examples of observed real-world driving behavior.

[0020] In an embodiment, object tracking may be applied to the sensor input to determine an observed trail of the external subject, and an expected behavior of the external subject may be simulated based on the observed trail.

[0021] At least some examples of observed real-world driving behavior used to train the generative model may have been extracted from CCTV data.

[0022] Generating the behavioral model may include a trained neural network.

[0023] The expected behavior of the external agent may be simulated using learned occupancy probabilities and / or transition probabilities of a spatial Markov model learned for the driving region of the encountered driving scenario.

[0024] The tree search algorithm may be a probabilistic tree search algorithm.

[0025] The tree search algorithm may be a Monte Carlo Tree Search (MCTS) algorithm.

[0026] The driving scenario may be determined by extracting a set of driving scenario parameters from the sensor input according to a defined scenario description language.

[0027] The generative behavior model is also applied to one or more ego vehicle parameters and / or one or more parameters of the driving scenario.

[0028] In a second aspect of the invention, the generative behavior model takes the form of an inverse planner that applies an inverse planning approach to one or more observed parameters to simulate the expected behavior.

[0029] That is, the second aspect provides an autonomous driving vehicle (AV) planning method, comprising:

[0030] receiving sensor inputs related to the AV;

[0031] Process sensor inputs to determine encountered driving scenarios;

[0032] In the AV planner, executing a tree search algorithm to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; and

[0033] generating an AV control signal for executing the determined AV manipulation sequence;

[0034] wherein the game tree has a plurality of nodes representing expected states of an encountered driving scenario, and an expected driving scenario state of each child node is determined by updating the driving scenario state of a parent node of the child node based on (i) a candidate AV maneuver and (ii) an expected behavior of at least one external agent in the encountered driving scenario; and

[0035] Therein, the expected behavior of the external agent is simulated by applying an inverse planning method to one or more observed parameters of the external actor derived from sensor input.

[0036] The inverse planning method may include determining a set of available goals for external actors in an encountered driving scenario, and applying probabilistic or deterministic goal recognition to the set of available goals in order to simulate expected behavior. Deterministic goal recognition means identifying one of the available goals as being implemented by the external actor. Probabilistic goal recognition means estimating, for at least one of the available goals, a probability that the external agent is implementing the goal (e.g., a probability for each goal that is the probability that the external actor is implementing the goal).

[0037] Object tracking may be applied to the sensor input to track at least one external actor in an encountered driving scene and thereby determine an observed trail of the external actor over a time interval.

[0038] Inverse planning methods may include:

[0039] Determining a set of available targets for external actors in an encountered driving scenario;

[0040] For each of the available targets, an expected trajectory model is determined;

[0041] comparing the observed trace of the external actor to a model of the expected trajectory of each of the available targets to determine the likelihood of that target; and

[0042] The determined likelihood of at least one of the targets is used to model expected behavior of the external actor.

[0043] The step of simulating the expected behavior of the external actor using the determined likelihood of the at least one target may include calculating at least one predicted trajectory of the external actor based on the expected trajectory model of the at least one target and the determined likelihood of the target.

[0044] The expected trajectory model for each target may be a distribution of predicted trajectories associated with that target.

[0045] Each target G i The distribution of includes: the conditional probability p(Τ|G i ), and the likelihood of the target p(G i |τ) is used to estimate at least one predicted trajectory probability p(Τ|τ) given an observed trace τ.

[0046] The observed traces may be used to predict a best available trajectory model for the target, and the comparing includes comparing the best available trajectory model to the expected trajectory model.

[0047] The observed traces may be used to predict current maneuvers and / or future maneuvers of the external actor, and the predicted current maneuvers or future maneuvers may be used to determine the best available trajectory model.

[0048] A sequence of a plurality of maneuvers may be determined for at least one target, and a best available trajectory model for the target may be determined based on partial trajectory models respectively associated with the plurality of maneuvers.

[0049] Each partial trajectory model may include one or more target motion values, and the one or more motion values ​​for a future portion of the best available trajectory model may be determined by applying motion smoothing to the target motion values.

[0050] The expected trajectory model for each target may be a single expected trajectory for that target.

[0051] The best available trajectory model for each target may be a single best available trajectory.

[0052] The partial trajectory model for each maneuver may be the most likely partial trajectory for that maneuver.

[0053] The defined cost function may be applied to both the expected trajectory model and the best available trajectory model for each target to determine respective costs of the expected trajectory model and the best available trajectory model, wherein the comparing includes comparing those costs.

[0054] The cost function can reward reduced driving time while penalizing unsafe trajectories.

[0055] The cost function also penalizes lack of comfort.

[0056] The expected behavior of an external agent can be simulated by sampling targets from a set of available targets based on the determined likelihood of the target.

[0057] Different target distributions may be determined for different maneuvers or sequences of maneuvers, and the expected behavior of an external agent may be simulated by sampling maneuvers or sequences of maneuvers from a set of possible maneuvers based on the maneuver distributions, and then sampling targets from a set of available targets based on the target distributions determined for the maneuvers or sequences of maneuvers.

[0058] Inverse planning at the maneuver level may be used to determine the maneuver distribution, which may include determining, for each of the possible maneuvers, an expected trajectory model, and comparing observed traces of external actors to the expected trajectory model for each of the possible maneuvers to determine the likelihood of the maneuver.

[0059] The search algorithm may be executed multiple times for multiple constructed game trees with different sampling targets, and the AV maneuver sequence may be determined by the AV planner based on statistical analysis of the paths within the multiple constructed game trees.

[0060] Object tracking may be applied to the sensor input to track at least one external actor in an encountered driving scene and thereby determine an observed trail of the external actor over a time interval;

[0061] Among them, the reverse planning method may include:

[0062] Determine the set of possible maneuvers against external actors in the encountered driving scenario,

[0063] For each of the possible maneuvers, determine the expected trajectory model,

[0064] comparing the observed trace of the external actor to the expected trajectory model for each of the possible manipulations to determine the likelihood of that manipulation, and

[0065] The determined likelihood of at least one of the maneuvers is used to simulate expected behavior of the external actor.

[0066] The expected trajectory model for each maneuver may be a single trajectory associated with that maneuver.

[0067] The expected trajectory model for each maneuver may be a distribution of predicted trajectories associated with that maneuver.

[0068] A third aspect of the present invention provides an autonomous vehicle (AV) planning method, comprising:

[0069] receiving sensor inputs related to the AV;

[0070] Process sensor inputs to determine encountered driving scenarios;

[0071] In the AV planner, executing a tree search algorithm to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; and

[0072] generating an AV control signal for executing the determined AV manipulation sequence;

[0073] wherein the game tree has a plurality of nodes representing expected states of an encountered driving scenario, and an expected driving scenario state of each child node is determined by updating the driving scenario state of a parent node of the child node based on (i) a candidate AV maneuver and (ii) an expected behavior of at least one external agent in the encountered driving scenario; and

[0074] Therein, expected behavior of an external agent is simulated by applying target recognition to one or more observed parameters of the external actor derived from sensor input.

[0075] Target identification may include computing, for each of the set of available targets:

[0076] the best available trajectory model for the target based on one or more observed parameters, and

[0077] Optimal trajectory model for the target.

[0078] Target identification can be probabilistic, and the target distribution can be determined by comparing the best available trajectory model for each target to the optimal trajectory model.

[0079] The defined cost function may be applied to both the expected trajectory model and the best available trajectory model for each target to determine the respective costs of the expected trajectory model and the best available trajectory model, and those trajectory models may be compared by comparing their costs.

[0080] Multiple target distributions may be determined for multiple possible maneuvers or sequences of maneuvers.

[0081] The expected behavior can be simulated based on targets sampled from a set of available targets based on the target distribution.

[0082] The expected behavior may be simulated by sampling at least one maneuver from possible maneuvers based on a maneuver distribution determined for the external agent, and then sampling a target from a target distribution determined for the sampled maneuver.

[0083] The tree search algorithm may be executed multiple times for multiple constructed game trees with different sampling targets, and the AV maneuver sequence may be determined by the AV planner based on statistical analysis of the paths within the multiple constructed game trees.

[0084] Alternatively, object recognition may be performed using an object recognition decision tree.

[0085] Another aspect of the present invention provides a method for configuring an autonomous driving vehicle, the method comprising:

[0086] At a training system, receiving real-world driving behavior data captured from one or more driving areas;

[0087] Processing real-world driving behaviors to extract examples of real-world driving behaviors for training the generative model;

[0088] Using the extracted driving behavior examples to train a generative behavior model to predict the behavior of the external subject based on one or more observed parameters of the external subject; and

[0089] The trained generative behavior model is stored in an electronic storage device of an autonomous vehicle, wherein the autonomous vehicle includes an autonomous vehicle planner configured to use the generative behavior model to implement the method of any aspect or any embodiment thereof disclosed herein.

[0090] The above-mentioned sensor input may include sensor input received from an AV sensor system.Further or alternatively, the sensor input may include externally captured sensor input received via a communication link.

[0091] Another aspect of the present invention provides an autonomous vehicle (AV) planning method, comprising:

[0092] receiving sensor inputs related to the AV;

[0093] Processing AV sensor inputs to determine encountered driving scenarios;

[0094] In the AV planner, executing a tree search algorithm to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; and

[0095] generating an AV control signal for executing the determined AV manipulation sequence;

[0096] The game tree has multiple nodes representing expected states of encountered driving scenarios, and the expected driving scenario state of each child node is determined by updating the driving scenario state of the child node's parent node based on (i) candidate AV manipulations and (ii) expected behavior of at least one external agent in the encountered driving scenario.

[0097] Any of the above features may be implemented in its embodiments.

[0098] Another aspect of the present invention provides a computer system comprising execution hardware and a computer program comprising executable instructions, wherein the execution hardware is configured to perform any of the method steps disclosed in this document, and the executable instructions are configured to implement any of the method steps when executed.

[0099] Yet another aspect provides an autonomous vehicle (AV) planner specifically in a computer system and an autonomous vehicle comprising the autonomous vehicle planner and a drive mechanism, wherein the autonomous vehicle (AV) planner is configured to implement any of the method steps disclosed herein, and the drive mechanism is coupled to the autonomous vehicle planner and responds to control signals generated by the AV planner. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to better understand the present invention and to illustrate how embodiments of the present invention may be implemented, please refer to the following drawings, in which:

[0101] Figure 1 A schematic functional block diagram is shown, which illustrates the functional components implemented in an autonomous vehicle computer system;

[0102] Figure 2An exemplary game tree that can be used for autonomous vehicle maneuver planning is shown;

[0103] Figure 3 shows a flow chart of a tree search algorithm that can be used to select a sequence of moves to be performed from a game tree;

[0104] Figure 4 A schematic functional block diagram of a training system for training a generated behavior model is shown;

[0105] Figure 5A-Figure 5C An example of the present technology implemented in a simulator is shown;

[0106] Figure 6 A flow chart showing the inverse planning method;

[0107] Figure 7A-7C Some principles of inverse planning are illustrated by examples;

[0108] Figure 8 An example of a tree search based on two possible manipulations is shown;

[0109] Fig. 9 An example of a tree search based on more than two manipulations is shown;

[0110] Fig.10 A flow chart showing an exemplary inverse planning method is shown;

[0111] Fig.11 Decision trees for alternative target recognition methods are shown;

[0112] Fig.12 A schematic functional block diagram of a maneuver planning method incorporating predictions from inverse planning is shown;

[0113] Fig.13 An example of a trajectory model learned from CCTV data is shown;

[0114] Figure 14A-Figure 14C shows exemplary scenarios in which the described techniques may be implemented;

[0115] Fig.15 A high-level flow chart is shown to illustrate certain principles of Monte Carlo tree search applied to probabilistic predictions about other(s); and

[0116] Fig.16 An example of a spatial Markov model is shown that models normal driving behavior in an area monitored by CCTV. DETAILED DESCRIPTION

[0117] The exemplary embodiments of the present invention are described in detail below. First, some useful background of the present invention is described.

[0118] Figure 1 A highly schematic functional block diagram of certain functional components implemented in an onboard computer system A1 of an AV (ego vehicle), namely a data processing component A2, a prediction component A4 and an AV planner A6, is shown.

[0119] The data processing component A2 receives sensor data from the AV's onboard sensor system A8. The onboard sensor system A8 can take different forms, but typically includes various sensors such as image capture devices (cameras), LiDAR units, etc., (one or more) satellite positioning sensors (GPS, etc.), (one or more) motion sensors (accelerometers, gyroscopes, etc.), etc., which collectively provide rich sensor data from which detailed information about the surrounding environment and the state of the AV and other agents (vehicles, pedestrians, etc.) within the environment can be extracted.

[0120] However, it should be noted that the present technology is not limited to using image data captured using the AV's own onboard optical sensors (image capture devices, lidar, etc.). Alternatively or additionally, the method may be applied using externally captured sensor data (e.g., CCTV images captured by an external image capture unit near the AV, etc.). In this case, the AV may receive at least some of the sensor inputs used to implement the method from the external sensor data source via one or more wireless communication links.

[0121] The data processing system A2 processes the sensor data to extract such information therefrom. This will typically involve various forms of machine learning (ML) / artificial intelligence (AI) processing. In the present context, the relevant data processing system A2 functions include localization (block A10), object detection (block A12), and object tracking (block A14).

[0122] Localization is performed to provide awareness of the surrounding environment and the AV's position within that environment. To this end, various localization techniques may be used, including visual and map-based localization. By way of example, reference is made to UK Patent Application No. 1812658.1 entitled "Vehicle localization," which is incorporated herein by reference in its entirety. This patent application discloses a suitable localization method using a combination of visual detection and predetermined map data. Segmentation is applied to the visual (image) data to detect surrounding road structures, which are in turn matched with predetermined map data (such as an HD (high definition) map) in order to determine an accurate and robust estimate of the AV's position in a map reference frame relative to the road and / or other structures of the surrounding environment, which are in turn determined by merging the visual and map data through a combination of visual detection and map-based inference. To determine a position estimate, individual position estimates determined from structure matching are combined with other (one or more) position estimates (such as, GPS) using particle filtering or the like to provide an accurate position estimate of the AV in a map reference frame that is robust to fluctuations in the accuracy of the individual position estimates. After the AV position on the map has been accurately determined, the visually detected road structure is merged with predetermined map data to provide a comprehensive representation of the vehicle's current and historical surroundings in the form of a real-time map and an accurate and robust estimate of the AV's position in the map reference frame. The term "map data" in the present context includes map data such as a real-time map derived by merging visual (or other sensor-based) detections with predetermined map data, but also includes predetermined map data or map data derived solely from visual / sensor detections.

[0123] Object detection is applied to sensor data to detect and locate external objects within the environment, such as vehicles, pedestrians, and other external actors, to which the AV needs to be able to safely respond. This may include, for example, a form of 3D bounding box detection, where the position, orientation, and size of objects within the environment and / or relative to the ego vehicle are estimated. This may be applied, for example, to (3D) image data such as RGBD (red green blue depth), LiDAR point clouds, etc. This allows the location and other physical characteristics of such external actors to be determined on a map.

[0124] Object tracking is used to track any movement of objects detected within the environment. The result is an observation trace (τ) for each object determined over time by object tracking. The observation trace τ is a history of moving objects that captures the path of the moving object over time and may also capture other information such as the object's historical velocity, acceleration, etc. at different points in time.

[0125] Used in combination, object detection and object tracking allow external actors to be comprehensively located and tracked on a map of the AV's surroundings.

[0126] Object detection and object tracking are well known per se and can be performed in the present context using various publicly available prior art models.

[0127] Through a combination of localization, object detection, and object tracking, the data processing component A2 provides a comprehensive representation of the environment surrounding the ego vehicle, the current state of any external actors within that environment (position, heading, speed, etc. within the range that can be detected), and the historical tracks of such actors that the AV has been able to track. This is continuously updated in real time to provide up-to-date position and environment awareness.

[0128] The prediction component A4 uses this information as the basis for a predictive analysis, wherein the prediction component A4 makes predictions about the future behavior of external actors in the vicinity of the AV. An example of a suitable prediction method is described below.

[0129] The AV planner A6 uses the extracted information about the self-surrounding environment and the external subjects within the environment together with the behavior predictions provided by the prediction component A4 as the basis for AV planning. That is, the predictive analysis performed by the prediction component A4 adds a layer of predictive information on top of the information extracted from the sensor data by the data processing component, which in turn is used by the AV planner A6 as the basis for AV planning decisions. This is typically part of a hierarchical planning process in which the AV planner A6 makes various high-level decisions and then makes increasingly lower-level decisions required to implement the high-level decisions. The end result is a series of real-time, low-level action decisions. To implement these decisions, the AV planner A6 generates control signals that are at least partially input to the AV's drive mechanism A16 to control the vehicle's speed and direction of travel, etc. (e.g., by steering, braking, accelerating, shifting gears). Control signals are also generated to perform auxiliary actions such as signaling.

[0130] The scene extraction component A3 uses the output of the data processing component A2 to determine the driving scene encountered by the ego vehicle. The determined driving scene includes driving scene parameters, which are extracted from the captured sensor data and provide a representation of the real world scene encountered by the AV that is concise but detailed enough to be used as the basis for realistic simulation. This is formulated in a structured scene description language, which can be used as the basis for such simulations. An example of a scene description in a formal scene description language is provided in Appendix A.

[0131] The simulator A5 receives the parameters of the encountered driving scenario and can run simulations based on these parameters. These are simulations of what might happen in the encountered driving scenario under different assumptions. These simulations are used as the basis for AV planning, where the AV planner A6 runs multiple simulations with the goal of determining the global optimal sequence of maneuvers to be taken in the encountered driving scenario to perform the defined goals (i.e., achieve the desired results, such as reaching a specific location on a map). In the example described below, the simulation is run as part of a Monte Carlo Tree Search (MCTS) performed by the maneuver selection component A7 of the AV planner A6.

[0132] The function of the prediction component A4 is to model the predicted external agent behavior to be run as part of the simulation. That is, the external agent behavior model is implemented to predict the behavior of any external actors in the encountered driving scenario so that the predicted behavior can be incorporated into the simulation on which the maneuver planning is based.

[0133] Operation Planning

[0134] As the ego vehicle is traveling, it is necessary to plan which maneuvers to perform in order to execute the defined goals. As indicated, this can be achieved using a Monte Carlo Tree Search, which is a heuristic search algorithm for decision making processes. The Monte Carlo Tree Search is an "online" reasoning process, i.e., it is performed by the AV planner A6 in order to make actual driving decisions in real time.

[0135] MCTS is applied to game trees. Game trees are data structures derived from game theory that provide a framework for making rational decisions via systematic analysis of different possible outcomes that can be meaningfully scored. A fully constructed game tree captures every possible outcome of a "game" with a finite number of possible "moves", where the term game is used in a mathematical sense to mean a formal model of strategic interactions (moves) between decision-making agents. Tree search algorithms can be applied to game trees, with the goal of determining a sequence of moves corresponding to a particular path through the game tree that is (globally) optimal with respect to a defined reward (score) function.

[0136] MCTS is an efficient probabilistic tree search algorithm for searching game trees. As different paths are explored, the game tree is dynamically constructed. The benefit of MCTS is that it can provide an asymptotically correct approximation without fully exploring every possible path through the game tree, i.e., without considering every possible sequence of moves, which in turn means that the game tree does not need to be fully constructed. In this case, MCTS does not guarantee that a true global optimal sequence of moves will be found, but the algorithm will converge towards a sufficiently optimized solution given a reasonable amount of time and computing resources.

[0137] Note that in the present context, there may be variations on a discrete set of ordered maneuvers that are considered a unique sequence for present purposes. For example, "following lane" between times t0 and t1 and "changing lanes" between t1 and t2 may be considered a different sequence of maneuvers than "following lane" between times t0 and t1' (different from t1) and changing lanes between t1' and t2, because the change from following lane to changing lane occurs at different respective times.

[0138] In the present context, movement is a driving maneuver. When the ego vehicle encounters an actual driving scenario that it needs to navigate, MCTS is used as a means to systematically reason about different possible outcomes of different ego maneuvers (i.e., maneuvers that the ego vehicle's AV planner A6 might take) in the encountered driving scenario, taking into account the predicted behavior of any external agents (other vehicles, pedestrians, etc.) therein. This is achieved by parameterizing the encountered driving scenario based on the captured sensor data as described above, and running different simulations in the simulator A5 based on the extracted driving scenario parameters and the external agent behavior as modeled by the prediction component A4.

[0139] The use of a probabilistic tree search algorithm (such as MCTS) allows finding sufficiently optimal maneuver sequences for large game trees as part of real-time AV decision making. In the current context, game trees can be large because there may be many agents in the scene.

[0140] The results of these simulations are used to evaluate the defined reward function in order to score different possible paths through the game tree, which in turn drives the execution of MCTS towards the global optimal path relative to the reward function.

[0141] Figure 2 An exemplary game tree D00 is shown in FIG.

[0142] The game tree D00 has a root node D02 representing the current state s0 of the encountered driving scene (defined for convenience as time t = 0). This captures the actual current state of the ego vehicle and any external agents within the encountered driving scene relative to their surroundings, in particular the current state (position, velocity, acceleration, etc.) relative to the surrounding road layout / structure.

[0143] Additional nodes of MCT D00 represent the expected (i.e., predicted, future) states of the driving scenario at different times in the future under different assumptions about the behavior of the ego vehicle. The structure of MCT D00 allows the AV planner A6 to reason about what might happen in the driving scenario if the ego vehicle were to perform a given sequence of maneuvers.

[0144] To facilitate this reasoning, MCT D00 is structured as follows.

[0145] Possible maneuvers are represented by the edges of the Monte Carlo tree D00. Suppose possible maneuvers of the ego vehicle given its current state s0 and a defined goal to be performed.

[0146] Each node except the root node D02 is a child of another node (its parent node) and represents the expected state of the driving scenario. The expected state of each such node is determined based on a simulation of an encountered driving scenario, which is run in the simulator A5 under a certain set of relevant assumptions, as will now be described.

[0147] The edges from each parent node to its corresponding one or more child nodes are shown. The edge between each parent-child node pair represents a corresponding maneuver that the AV planner A6 can take given the state of the driving scene represented by the parent node (at time t); the child node in turn represents the expected state of the driving scene (at time t+Δt) if the maneuver is performed over an interval Δt.

[0148] The predicted driving scenario state of each child node is determined by updating the driving scenario state represented by its parent node based on an appropriate simulation of the driving scenario. The driving scenario simulation is run based on: the extracted driving scenario description parameters; the parent state (which can be used, for example, as the starting state for the simulation); the corresponding maneuvers between the parent node and the child node (whose execution is simulated between time t and t+Δt); and the simulated external subject behavior between time t and t+Δt (as modeled by the prediction component A4).

[0149] The execution of the corresponding maneuver by the ego vehicle in the interval ΔT is simulated by simulating or “showing” the actions that the AV planner A6 would take in real life given the state of the driving scenario represented by the parent node, the external agent behavior in that time interval ΔT, and the maneuver to be performed.

[0150] The execution of a maneuver of the ego vehicle can be simulated using an "action policy" that has been learned or otherwise determined to perform the maneuver in question given a particular state. The action policy is also used by the AV planner A6 as the basis for performing the maneuver in the real world. For example, the action policy can be learned "offline" by reinforcement learning (RL). For example, reference is made to British patent application No. 1816852.6 entitled "Autonomous Vehicle Manoeuvres", which is incorporated herein by reference in its entirety. This patent application discloses a framework for maneuver learning, in which training scenarios are determined from observed driving behavior data.

[0151] As indicated, external agent behavior is simulated by executing an external agent behavior model of prediction component A4. Examples of suitable behavior simulation models are described later.

[0152] For example, Figure 2 Edges D04a-c are shown from the root node D02 to the root node's three direct child nodes, which are labeled D06a-c. Each of these edges D04a-c represents a different maneuver that can be performed given the state s0 of the driving scene from the node from which they extend (the root node D02 in this example). Each of these child nodes represents a subsequent state s0 of the driving scene, respectively. 1A 、s 1B 、s 1C , the subsequent states are predicted if the ego vehicle performs the corresponding maneuver starting from state s0.

[0153] For example, with respect to the parent state s0, in a multi-lane driving scenario, edges D04a-c may correspond to "following lane", "left turn lane", and "right turn lane", respectively. In state s0, the ego vehicle is in the current lane: following lane maneuvers are performed to stay in the current lane within the applicable time interval Δt; left turn lanes and right turn lanes are performed to try to move to the lanes to the left and right of the current lane, respectively. 1A 、s 1B 、s 1C is obtained by advancing the parent state s0 according to the relevant manipulation by taking into account the external subject behavior over the relevant time interval.

[0154] Other exemplary maneuvers that may be applicable include Slow Down, Continue, and Exit (eg, when navigating a roundabout).

[0155] As will be appreciated, the driving scenario states of nodes further down the tree are obtained in exactly the same way, starting from the node's corresponding parent state.

[0156] Some nodes are terminal nodes, that is, nodes that have no child nodes, such as Figure 2 Nodes D08 and D10 in . These represent "terminal states," which, broadly speaking, correspond to a point at which the ego vehicle has successfully executed a defined goal (success), or to a point at which it is determined that the defined goal has failed. Failure can be defined in various ways depending on the context, but examples include: the ego vehicle is forced to abort the goal; the vehicle is determined to have made insufficient progress; and the vehicle crashes or otherwise fails for safety reasons. Depending on the context, starting from the root node D02, a maneuver or sequence of maneuvers may be required to reach the terminal node.

[0157] Optionally, the end goal can be decomposed into two or more intermediate goals, which may be appropriate when the end goal is too far away. In this context, each intermediate goal can be executed based on its own game tree, which has a terminal node representing the end state of the intermediate goal (but not necessarily the end state of the end goal).

[0158] Game tree D00 is a finite tree because every possible path through game tree D00 eventually terminates at a terminal node after a finite number of moves.

[0159] Based on the reward function, a score (reward) is assigned to each considered terminal node, which indicates the desirability of the result represented by it. This can be a simple binary scheme, with two scores for success and failure respectively, or a richer scoring mechanism can be applied when appropriate. For example, if a successful result is reached in a shorter amount of time, or a successful result is otherwise more desirable under any standard appropriate in the situation, the successful result can be scored more highly. The score assigned to each possible path through the game tree is simply the score assigned to its terminal node.

[0160] Rewards are not necessarily given only in terminal states. Each considered path is assigned a score (reward) based on a reward function that indicates the desirability of the outcome it represents. This can be a simple binary scheme, with two scores for success and failure, respectively, or a richer scoring mechanism can be applied where appropriate. For example, a successful outcome may be scored higher if it is reached in a shorter amount of time, or if a successful outcome is otherwise more desirable under any criteria appropriate in the context.

[0161] In this example, the game is a single-player model where each edge corresponds to a maneuver of the ego vehicle. The behavior of the external agent is captured as part of the state change that occurs when moving from an applicable parent node to an applicable child node. While this approach does demonstrate what other agents might do, it does not necessarily include in that demonstration how other agents might respond to hypothetical actions of the ego vehicle (but neither is excluded). That is, while the generated behavior can model the response of the external actor to the ego vehicle and to its surroundings (the driving context, such as the road layout), it is not required to model the high-level decisions made by the external actor, as acceptable results can be achieved with simpler models. However, the present technique can be extended to incorporate models of high-level planning (such as inverse planning) of external actors.

[0162] Only possible (ie, realistically achievable) maneuvers can be assumed. For example, if the vehicle is in a particular position relative to a T-intersection (corresponding to a parent state), there may be three possible maneuvers: stop, turn left, and turn right, but continuing straight will not be an option.

[0163] As another example, in a multi-lane road, the available maneuvers may be following lane and changing lanes. MCTS is used to sequence those maneuvers so that the vehicle remains in the following lane maneuver until it is safe to initiate the lane changing maneuver.

[0164] The model can incorporate a certain level of understanding of how vehicles behave in the real world. For example, it knows that one limitation of a vehicle is that it cannot turn on the spot. Therefore, it does not assume any maneuvers that would require impossible vehicle behavior.

[0165] One way to identify the optimal path through the game tree D00 is to evaluate the score of each possible terminal node and select the path with the highest score. However, in practice, this may not be feasible, especially in real time.

[0166] In this context, a probabilistic tree search such as MCTS can be used, which is able to find a path that is at least close enough to optimal in a reasonable amount of time using a reasonable amount of computational resources.

[0167] Figure 8 A simple example of MCT 800 is shown, which can be used when there are only two possible maneuvers for a given goal. For example, the goal may be to join the flow of traffic from a side road in a particular direction. In this example, the AV needs to turn left to join the road. If both roads are single lane, the two possible maneuvers that the AV can implement are stop or turn left. This results in a variety of possible maneuver sequences depending on when the vehicle changes from "stop" to turn left, e.g., (stop, stop, turn, turn, ...) versus (stop, stop, stop, turn, ...), etc.

[0168] The current state s0 of the AV is the state when the AV approaches the intersection. This is represented by the root node 802 at time t0. The AV can stop at the intersection or turn left onto the road. These two possible maneuvers are represented by edges 804a and 804b, respectively.

[0169] To achieve the goal of joining the main road, the AV must implement a left turn maneuver, edge 804b. However, this can be implemented at any time. MCT 800 shows the predicted state of the AV after each of five time intervals Δt, but it should be understood that more or fewer time intervals can be considered. Although the time intervals for determining the state are in Figure 8 The MCT is shown to be the same, but it should be understood that the time interval between each iteration can be changed.

[0170] The AV starts in state s0. If the AV chooses to turn left immediately onto the main road, following edge 804b, the predicted state is s 1B , represented by child node 806b. This is the terminal node because the AV has joined the main road. If state s 1B is a safe state for AV, then the termination node 806b is successful. However, if the state s 1B Unsafe, for example, the AV has collided with or is close to an external subject, then the termination node 806b is a failure.

[0171] Alternatively, at t0, the AV may perform a stop maneuver, following edge 804a. Thus, the resulting predicted state s 1A will correspond to an AV stopped at the intersection with the main road.The AV must then decide at time t0+Δt whether to implement a left turn maneuver (take edge 804b) or remain stopped (take edge 804a).

[0172] Again, this results in having a state s corresponding to the AV on the main road 2B and a child node 810a which is not a terminal node because the AV has not yet joined the main road.

[0173] This is repeated for each additional time step, where the path terminates in a state corresponding to the AV having joined the main road. Some of these states will be failures, while others will be successes. The optimal path is then found from the constructed MCT.

[0174] Fig. 9 An exemplary MCT is shown of an AV approaching a roundabout from a road having two lanes. Figure 5A-Figure 5C Such a scenario is shown (see below).

[0175] The goal is for the AV to join a roundabout. The initial state s0 includes the AV approaching the roundabout in the left lane. The AV can perform three possible maneuvers: stop, follow lane, and turn right. These three maneuvers are represented by edges 904a, 904b, and 904c, respectively. The resulting state s 1A 、s 1B and 1C These correspond to the AV stopping at the intersection with the roundabout, joining the roundabout in its current lane, or switching to the right lane of the road, respectively.

[0176] If AV follows edge 904b, the resulting state s 1B It is the termination node 906b. This is because the AV has already joined the roundabout. The termination node 906b may or may not be a success node.

[0177] If the AV performs a right-moving maneuver, following edge 904c, the AV enters the right lane of the road. The AV then has the option of continuing forward (edge ​​908b) or stopping (edge ​​908a). By continuing forward, the AV joins the roundabout, so the path ends with the ending node 910b. However, if the AV stops, the AV can perform a maneuver to stop or follow the lane (i.e., join the roundabout in the right lane) at the next time interval.

[0178] If the AV performs a stop maneuver (following edge 904a) from the current state s0, the AV has the option to perform any one of the three maneuvers, ie, stop, follow lane, and move right, again.

[0179] Each time the following lane maneuver is implemented, a termination node is generated because the AV has already joined a roundabout. When a move right maneuver is implemented, the move right maneuver cannot be implemented at the next iteration, so the number of child nodes is reduced by one. When a stop maneuver is implemented, all three maneuvers remain feasible. The MCT can be constructed for all possible paths in this way. Once the MCT is constructed, the optimal path has been found.

[0180] As mentioned above, the MCT does not need to be fully constructed in order to find a sufficiently optimal solution.

[0181] Figure 3 It shows that it can be Figure 2 Flowchart of an exemplary MCTS algorithm executed on the game tree D00 of . Because the MCTS method itself is known, only the relevant steps are briefly described in order to illustrate the application of MCTS in the current context.

[0182] MCTS operates iteratively based on a probability calculation that can be thought of as a form of "back-propagation" of the reward from the terminal node under consideration (note that this is a different probability calculation than the chain rule used for back-propagation in neural networks).

[0183] The game tree D00 is built dynamically as MCTS executes, and in practice it may only be partially built. A node is added each time a new state is encountered, and possible maneuvers that the vehicle can perform from that state are hypothesized. This continues until a terminal node is reached.

[0184] Each iteration starts with traversing the existing tree, starting from initially selecting the root node D02 (step E0). The selected node is marked as visited (E2), and a single path from the root node to the terminal node is attempted through the game tree D00 (E4). This means that the simulation is run to determine the continuous change of the state of the driving scene, as described above, (starting from the state of the root node) until the terminal node is reached. The expansion is random, that is, a random selection of manipulations and the addition of new nodes to the tree. If the new node is not a terminal, a rollout is simulated for the terminal node. The rollout is also random, that is, starting from a leaf node, a random selection of manipulations is made to progress to the terminal node from there. Then, the iteration is scored as described above, and the information about the score is back-propagated (E6) through the visited nodes. Next, an unvisited node is selected (E8), and the process is repeated for the newly selected node starting from step E2.

[0185] Each iteration may start by initially selecting the root node D02 by "visiting" a node that has not been visited before (step E0). The selected node is marked as visited (E2), and a single path from the selected node to the terminal node through the game tree D00 is expanded (E4). This means that the simulation is run to determine the continuous change of the state of the driving scene, as described above, (starting from the state of the selected node) until the terminal node is reached. The path extension is random, that is, starting from the currently selected node, a random manipulation is selected to progress from there to the terminal node. Then, as described above, the terminal node is scored, and information about the score is back-propagated through the expanded path (E6) (note: nodes along the expanded path are not marked as visited). Next, an unvisited node is selected (E8), and the process is repeated for the newly selected node starting from step E2. The unvisited nodes are selected based on the score information that has been back-propagated through the game tree so far, so that (over time) the selection of nodes is biased towards more promising results, while still encouraging a sufficient level of exploration to mitigate the risk of unwanted convergence to a mere local optimum.

[0186] The process is repeated iteratively until it terminates (E10). This may be, for example, when a certain time limit has been reached or a certain number of iterations have been performed, etc. The benefit of MCTS is that it can be terminated at any time, although the longer it runs, the closer it will get to the average global optimal solution.

[0187] Upon termination, the AV planner A6 selects one of the expansion paths determined to be the most promising and generates a control signal for controlling the AV to perform a corresponding maneuver sequence in the real-world driving scenario that has been encountered ( E12 ).

[0188] The most promising path may be the path with the largest score. However, information other than scores may also be considered (see below).

[0189] In real driving scenarios encountered by AVs, the road layout will typically be fixed and known, however other vehicles / actors in the scene are dynamic. As such, it is possible that new actors will be detected as the MCTS process runs over time. When new actors are detected during an ongoing MCTS process, the structure of the constructed game tree including existing nodes and edges that have already been constructed is retained. However, the simulation is rerun on the existing game tree in order to recalculate the reward function taking into account the simulated behavior of the newly detected agent.

[0190] Figure 5A An example of the MCTS process applied to a roundabout driving scenario in a simulator is shown. The ego vehicle is represented by reference numeral 500 and needs to plan safely in a roundabout scenario in the presence of other vehicles 502 on the roundabout. This is done by selecting a sufficiently optimal sequence of maneuvers of "slow down", "follow lane" and "switch to right lane" (right switch).

[0191] Figure 5B Shows how different manipulation sequences are evaluated by MCTS.

[0192] Figure 5C The ego vehicle 500 is shown selecting and executing a maneuvering sequence to enter a roundabout. The process then repeats to determine and execute a maneuvering sequence to exit the roundabout.

[0193] As described below, MCTS can be implemented with probabilistic predictions about the behavior of other agents. For example, both probabilistic manipulation detection and probabilistic target recognition are considered below. Using probabilistic predictions, one or more relevant distributions are sampled at the beginning of the process to effectively obtain a deterministic prediction or set of predictions ((one or more) samples). MCTS is then repeated using different deterministic predictions sampled from the probabilistic predictions.

[0194] Fig.15 A high-level flow chart of MTCS implemented using probabilistic prediction is shown.

[0195] In step 1502, at least one probability prediction is determined for the external actor, which may be, for example, a distribution P(M|τ) over possible manipulations of the agent P(M|τ) and / or a distribution P(G|O) over possible goals P(G|O) (see below).

[0196] Symbol O represents a set of observations. Observations O may be the observed traces τ themselves (O=τ), but O is not necessarily limited to this aspect. For example, in one embodiment, observations O may include the subject's manipulations M j In this case, the notation P(G|M j ) (possibly as a shorthand, since the observation O may include (one or more) additional parameters) (note also that the manipulation can be inferred from the trace τ).

[0197] In step 1504, the distribution(s) of step 1504 are sampled to obtain a set(s) of deterministic predictions. For example, P(M|τ) may be sampled to select the actual current maneuver M j And / or P(G|O) may be sampled to select the actual current target in a statistically robust manner (making more likely targets, maneuvers, etc. more likely to be selected).

[0198] In step 1506, the MCTS process is performed in its entirety as described above. This can be iterated as a "super iteration" of the MCTS process to distinguish it from the iterative steps of exploring the tree performed within the MCTS process.

[0199] Multiple super-iterations of MCTS are repeated with different samplings of the probability predictions, e.g., different goals / maneuvers sampled from the same goal / maneuver distribution P(G|O) / P(M|τ) in step 1504. That is, the probability prediction(s) are resampled for each super-iteration in step 1504. This repetition is shown as being performed until a stop condition set (1508) is met, e.g., a time limit is reached, a certain number of super-iterations are performed, etc.

[0200] After all super iterations of the MCTS process are completed, a statistical analysis of all results is applied in step 1510 to select the most promising maneuver sequence for the ego vehicle.

[0201] The statistical analysis of the overall results in step 1510 takes into account the scores determined in each super iteration of MCTS. However, other information may also be considered, such as the "confidence" associated with a given manipulation sequence. The confidence reflects the extent to which the manipulation sequence has been tested via simulation. For example, a particular manipulation sequence may be highly scored in one MCTS super iteration, but just not tested in other super iterations. In that case, the sequence has a high score but a low confidence, and can be ignored in favor of a lower scoring manipulation sequence with a higher confidence, i.e., a manipulation sequence that has been more extensively tested.

[0202] External subject behavior simulation

[0203] When constructing the game tree D00, in order to determine the driving scene state changes, it is necessary to predict the future movement of the external actors. As indicated, this external subject behavior is modeled by the prediction component A4 for this purpose.

[0204] Different forms of generative models can be used in this context.

[0205] An example of a suitable model is an "inverse planner". For example, reference is made to UK Patent Application No. 1816850.0 entitled "Autonomous Vehicle Planning and Prediction", which is incorporated herein by reference in its entirety. This patent application discloses an inverse planner that simulates the behavior of an external agent that can be used in the present context. Inverse planning is described below.

[0206] It can be assumed that the principal planning is at the same level as the vehicle, although not necessarily using the same methodology.

[0207] For prediction purposes, the knowledge shared by the vehicle and the external agent is assumed to be a map.

[0208] Further details of the specific mechanisms by which MCTS for steering planning may be combined with goal recognition are described later, following a description of inverse planning.

[0209] The reverse planning component A24 is shown as forming Figure 1 A part of the prediction component A4 in . The reverse planning component A24 (reverse planner) implements "reverse planning" as described later.

[0210] Additionally or alternatively, the generative model may be trained to predict the future movement of a subject using real-life driving behavior data such as CCTV (closed-circuit television) footage.

[0211] One such example is a neural network based model (or other trainable function approximator) that uses observed parameters (one or more) of an external agent (such as observed traces) in a short period of time before t0 to predict how the agent is likely to move. This can be a probabilistic model or can be deterministic. Such a model can be used to generate realistic behavior in driving contexts where the model has not been encountered before (such as unknown road layouts), thereby exploiting the ability of such models to generalize from training examples when properly trained.

[0212] Another example is a model of a known driving area, derived by direct observation of the driving area, e.g. based on CCTV footage. For example, in an urban driving context, models may be built for difficult driving areas such as complex / busy intersections, roundabouts, etc. One such model is a spatial Markov model, where the driving area is divided into grid cells and the occupancy probabilities of the grid cells and / or the transition probabilities between grid cells are determined by long term observation, and the model in turn may be used to predict external agent behavior based on its observed parameter(s).

[0213] These are examples of trained generative models of behavior.

[0214] See also Figure 4 , a generative model F06 can be trained to simulate external subject behavior offline in a training computer system F00. Real-world driving behavior data F01 (such as CCTV data) is received at a behavior extraction component F02 of the training system, which processes the data to extract examples of driving behavior for use in training. The training component F04 uses these examples to train a generative model F06 to simulate external vehicle behavior based on observation parameters (such as observation traces) as described above. The training component F04 and the behavior extraction component F02 are functional components of the training system F00 executed on one or more processors (not shown) of the training system F00. After the generative model F06 is trained in this manner, the generative model F06 can be incorporated into the AV by transferring the generative model F06 to the electronic memory of the on-board computer system A1 of the autonomous vehicle to enable the prediction component A4 to perform the above functions.

[0215] Reverse Planning

[0216] To assist the AV planner A6 in making AV planning decisions, such as determining the sequence of maneuvers, actions, etc., to be taken by the ego vehicle to safely and efficiently execute the defined goals, the inverse planner A24 predicts the movements of nearby external actors, which in the present context may be referred to as agents. The inverse planner A24 is one example of a generative behavior model that may be applied in the above context.

[0217] Inverse planning can be implemented at the maneuver level to make probabilistic predictions about the current maneuver of the external actor. For example, given a set of relevant observations including (or derived from) a trace τ, the inverse planner A24 can predict a probability distribution P(M|τ) over a set of available maneuvers M (such as "follow lane", "change lane", "change lane", etc.). Inverse planning at the maneuver level is a form of (probabilistic) maneuver detection.

[0218] Alternatively or additionally, inverse planning may be implemented at the goal level to make probabilistic predictions about the current goal of the external actor. For example, the inverse planner A24 may predict a probability distribution P(G|O) over the set of available goals G. For example, in a driving scenario with a left turn, the goal may be a "turn left" goal or a "continue straight" goal (i.e., stay on the current road and do not make a left turn) that is captured as a suitable target location. Inverse planning at the goal level is a form of (probabilistic) goal recognition.

[0219] Target recognition and manipulation detection will typically operate on different time scales. Target recognition typically considers longer time periods in the future than manipulation detection. For example, manipulation predictions may be a few seconds in the future (e.g., 5s order), while target recognition may (depending on the context) be further out than this. Therefore, target recognition will typically consider longer trajectories (i.e., trajectories further into the future) than manipulation recognition.

[0220] The goal may be captured, for example, as a desired location (reference point) on a map (i.e., in a map reference frame) that the ego vehicle is attempting to reach from its current location on the map, where the desired location is defined relative to the road layout encountered. For example, the desired location may be defined relative to a particular intersection, lane layout, roundabout exit, etc. Given the surrounding road layout and any external actors, there will be various paths / actions that the vehicle can take in order to successfully execute the goal by reaching the desired location. There are also ego vehicle actions that will prevent the ego vehicle from doing so, for example, selecting a poor initial sequence of actions may result in the vehicle being forced to take an unwanted roundabout exit, or aborting the goal when it is unsafe to continue executing the goal due to the actions of external actors. Incorporating reliable predictions about the behavior of external actors into planning helps the AV planner A6 plan safely and efficiently, thereby minimizing the occurrence of aborted goals.

[0221] When implemented at the goal level, inverse planning assumes different possible goals for each external agent, and then generates trajectories of how the agent can achieve each goal, as well as the likelihood that the agent will follow each trajectory. The underlying assumption is that each external agent will act in a way that can be predicted using a generative model.

[0222] Inverse planning in the present context refers to a specific class of prediction methods that may be implemented by the AV prediction component A4. That is, inverse planning methods are specific methods of predicting the behavior of external actors (particularly other vehicles) by assuming that the external actors plan in a predictable manner.

[0223] The term "inverse planning" refers to this underlying assumption that the external actor will plan its decisions in a predictable manner. More formally, the assumption is that the other vehicle will plan and execute with a generative model that can be assumed.

[0224] Now we will see Figure 6 To describe the inverse planning method, Figure 6 A flow chart of the method is shown. This considers inverse planning at the target level, but the basic principles apply equally to inverse planning at the maneuver level. The steps of the method are repeatedly performed by the inverse planner A24 in real time or pseudo real time, so that sufficiently up-to-date predictions are always available to the AV planner A6. The method makes use of information provided by the data processing system A2, i.e. information about the surrounding environment / road layout, the position / state of any other agents within the environment, and the traces of those (one or more) agents observed by object tracking.

[0225] The following steps are performed for each of the one or more external actors (vehicles, pedestrians, cyclists, etc.) under consideration, which are vehicles other than AVs in the following examples.

[0226] In step SB2 ( Figure 6 ), a set of hypothetical goals is determined for the other vehicle in question. It is assumed that the other vehicle is currently executing one of these goals. In order to determine an appropriate set of hypothetical goals, a driving context such as the road layout near the other vehicle is determined.

[0227] External subject goals are often based on map assumptions. For example, given a set of external vehicles in the vicinity of a road intersection, roundabout, or other road layout (driving context) indicated on a map, a suitable goal can be assumed from the road layout alone (without considering any observed historical behavior of the subject). For example, in the case of a set of external subjects in the vicinity of a left-turn intersection, the assumed goal may be to turn left and continue straight. As indicated, such a goal is defined with reference to a suitable target location (e.g., a reference point) on the map.

[0228] However, available targets can be assumed in various ways. For example, observed historical behavior (such as traces observed before time t) can be taken into account in assuming external subject targets, or a combination of map-based and historical behavior-based inferences can be used to assume targets.

[0229] Note that even when historical behavior is not used to assume available goals, it is still used to determine the likelihood of each of those goals (since P(G|O) depends on the observed value O of the agent's behavior).

[0230] Having identified a hypothetical set of targets, for each of these targets the following steps are performed:

[0231] In step SB4, an expected trajectory model is determined for the hypothetical target in question. The expected trajectory model is a model that simulates the future behavior of the other vehicle assuming that the other vehicle is executing the specific target. In particular, assuming that the other vehicle executes the target during the time period Δt, the expected trajectory model indicates the likelihood that the other vehicle will take one or more specific paths (trajectories) within a given time period Δt (from time t to time t+Δt). As indicated, the target that the vehicle is executing can be parameterized by an endpoint based on a map. For example, if the goal is to continue straight (rather than, for example, turning left), the endpoint can be a point on the road at a set distance (such as 40m) in front of the vehicle in the same lane. Alternatively, in, for example, a multi-lane scenario, the target position can be a certain distance along the road ahead without specifying a specific lane (see below for further details).

[0232] The expected trajectory model can be just a (single) predicted path for a given target, but in this example it takes the form of a distribution of predicted paths for the target in question. In the present case, given the position r of the external vehicle at time t, t In the case of , a discrete set of n predicted paths is synthesized for the target in question, thereby providing a predicted path distribution for at least the time interval Δt to provide a predicted path distribution for each target.

[0233] For example, Fig. 7A The corresponding predicted path sets P for targets G1 and G2, respectively, as predicted for a specific external vehicle are shown. G1 , P G2 The targets G1 and G2 are defined relative to reference positions R1 and R2 in the map reference system, respectively. The reference positions can be points or areas on the map. Given the position r of the external vehicle at time t, t and the reference points R1 and R2 of each target, respectively, the synthetic prediction path set P for target G1 and G2 G1 , P G2 .

[0234] Although in this example each target is simply defined relative to a single reference point / area, it will be appreciated that targets may be defined in other ways, such as with reference to multiple reference points where appropriate in the context. Typically, targets are defined by one or more sets of target parameters, which in turn are typically defined in a map reference system in the context of autonomous driving. A reference position in a map reference system is an example of a target parameter, and all descriptions relating to such a reference position apply equally to other types of target parameters.

[0235] Following the left turn example, a set of paths will be generated for the "continue straight" goal, one set of paths is the path that another vehicle is predicted to take when executing the "continue" goal, and another set of paths will be generated for the "turn left" goal, another set of paths is the path that another vehicle is predicted to take when executing the "turn left" goal.

[0236] These paths can be synthesized using a generative model. The underlying assumption is that another vehicle will plan and execute with this model. This model can correspond to the AV planner A6 itself (assuming that the other vehicle will plan in the same way as the ego vehicle), but it can also be different from the AV's own planner.

[0237] For example, a Rapidly Exploring Random Tree (RRT) model can be used to synthesize pathways for each target. Fig. 7A In the example of , for each target G1, G2, a space of predicted paths (search space) is defined based on the reference position of the target (R1, R2, respectively) and the current position r0 of the external vehicle. The search space is then randomly sampled (based on the randomized input parameters) to determine a set of n paths and the likelihood of each of these paths. To simulate n paths to each target, the relevant parameters of the RRT are randomized n times to perform n appropriately biased random searches of the search space.

[0238] For example, UK Patent Application No. 1803292.0, entitled "Efficient computation of collision probabilities for safe motion planning", discloses an RRT model that can be used in the present context to implement inverse planning, and the patent is incorporated herein by reference in its entirety. The collision probability risk along a given trajectory is calculated, and the collision probability risk is used to rank candidate trajectories according to safety. This in turn provides a likelihood for each sampled path, assuming that an external vehicle is more likely to take a safer path to perform the objective in question. That is, the path probability can be determined based on an assumed relationship to safety. The sampled path and its probability are an example of a trajectory model.

[0239] However, this is only one example of a suitable generative model, and other forms of generated models may also be used. Examples of alternative trajectory models are described later.

[0240] One such example is a neural network based model that has been trained to output a path prediction model (e.g., a predicted path or a distribution of predicted paths) given an observed trace and a goal to be performed. The network is trained based on examples of real-world driving behavior. For example, the neural network can be trained based on examples extracted from a large corpus of CCTV (closed-circuit television) data that has been captured in an urban driving environment.

[0241] Another example is a spatial Markov model (or similar), which can be predetermined for a specific driving area by monitoring the driving area over a sufficient period of time (see above).

[0242] In general, the inverse planner A24 can be any model that can reason about different hypothetical goals.

[0243] In step SB6, the track of another vehicle actually observed over time period Δt (ie, between time t and t+Δt) is matched with the distribution of paths associated with the target in question over that time period Δt to determine the likelihood of that target.

[0244] For example, Figure 7B The actual observed track τ of the vehicle in question between time t and t+Δt is shown. By comparing the actual track τ with the Fig. 7A ), the likelihood of each target G1, G2 can be determined probabilistically for the time interval Δt. This can be a form of soft matching. Given the observed trace τ of the external vehicle, the target likelihood can be captured as i The conditional probability, that is, p(G i |τ), which is the conditional probability that the external vehicle will execute the target G at the time interval Δt given the observed trace τ. i The estimated probability of .

[0245] In other words, the inverse planner A24 is used to predict for each hypothetical goal the set of possible paths that the other vehicle may have taken in the time interval Δt and the likelihood of each of these paths, assuming that the other vehicle has performed the goal during the time period (i.e., the operations that the other vehicle may have performed during the time interval Δt if the other vehicle has performed the goal). This is then compared with the actual trace of the other vehicle during the time period (i.e., the operations that the other vehicle actually performed) to determine the likelihood of each goal for the time period Δt.

[0246] refer to Figure 7C, now that the likelihood of each target given the observed trace τ is known, the (absolute) likelihood of any given predicted path (trajectory) Τ after time t+Δt given the observed trace τ can be determined, for example, as:

[0247]

[0248] Among them, p(G i |τ), and p(Τ|G i ) is given at time t+Δt for the target G i The estimated probability that the subject takes path Τ given the path distribution. Note that the symbols "Δt" and "ΔT" are equivalent in this paper. Also note the difference between the mathematical symbol used to represent trajectory (lowercase or uppercase Greek letter "tau", i.e., τ, Τ) and the symbol used to represent time (lowercase or uppercase Latin "t", i.e., t, T).

[0249] Trajectories may be simple spatial paths, but the description applies equally to trajectories containing motion information (such as speed / velocity information, acceleration). For the latter, two trajectories may correspond to the same spatial path (or at least corresponding portions of the trajectories may correspond to the same spatial path), but still be different trajectories, for example, because they are associated with different velocity information (for example, one velocity information may correspond to a vehicle moving along substantially the same spatial path but at a lower velocity (at least at some points along the spatial path)). For example, a trajectory may take the form of a spatial path combined with an associated velocity or set of velocities associated with corresponding points or portions of the spatial path. Extensions may additionally incorporate associated (one or more) acceleration values, etc.

[0250] Although not shown in the figure, at time t+Δt (from which p(Τ|G i The path distribution of the target G can be obtained by, for example, i Re-execute the inverse planner A24 at time t+Δt to determine, but with external principal r at time t+Δt t+Δt The new determined position of the vehicle is determined so that the expected trajectory model is updated accordingly. To re-execute the inverse planner A24, the target parameters may or may not have been updated at this time. In the case of using a data-driven behavior model, the behavior model may similarly be re-executed at this time to update the expected trajectory model.

[0251] The above steps are repeated over time, possibly in real time. For an external agent that is somewhat off from reaching its goal, it may not initially be possible to determine exactly which goal it is pursuing, because the distribution of paths to different goals is initially similar, and this will be reflected in the distribution of probabilities over the set of hypothesized goals. As the path distributions diverge, the probability distribution will generally begin to bias toward a particular goal as the path distributions diverge.

[0252] Inverse Planning for Manipulation

[0253] Above we consider inverse planning of the goal. As indicated, on a hypothetical set of maneuvers M, inverse planning can also be implemented at the maneuver level.

[0254] For example, if another vehicle is currently traveling on a multi-lane road with no intersections nearby, the hypothetical maneuver set M may include "following lane" and "changing lane". In this case, a path set will be generated for the "following lane" maneuver, which is a path set that is predicted to be taken by the other vehicle when the "following lane" maneuver is currently being performed, and another path set will be generated for the "changing lane" maneuver, which is a path set that is predicted to be taken by the other vehicle when the "changing lane" maneuver is being performed. The above description is also applicable to maneuver-level inverse planning, but the hypothetical target set G is replaced by the hypothetical maneuver set M.

[0255] For manipulation-level inverse planning, the observation trace τ of the external agent can be used to estimate each manipulation M in the set of possible manipulations M. j ∈M. For each manipulation M j ∈M, determine the predicted trajectory model at time t, such as Fig. 7A The predicted trajectory model can be a single trajectory or a trajectory distribution. Then, by combining the actual observed trace τ with each manipulation M j The trajectory model is compared (such as Figure 7B As shown), based on the observed trace τ and the control M j The probability of the maneuver is estimated by the degree of matching of the trajectory model. j |τ). As mentioned above, this can be a form of soft matching.

[0256] In the following example, to simplify the calculation, for the purpose of inverse planning, each maneuver M j is mapped to a single trajectory. For maneuvers associated with a distribution of trajectories, the mode or most likely trajectory of the distribution can be taken for the purpose of inverse planning. In this case, it is assumed that there is a single trajectory T j , so that:

[0257] P(T j |M j)=1

[0258] And all other trajectories are assumed to have zero probability.

[0259] The symbols used above are used in conjunction with the given manipulation M j The associated trajectory j and relative to a given target G i A distinction is made between the evaluated trajectories Τ.

[0260] To reach the target G i , an external subject can perform a sequence of operations, for example, (M j , M k , …). In this case, the overall target trajectory (i.e., reaching the target G i ) can be determined as being related to each control M i ,M j , ... associated with the control trajectory. For example, in the embodiment described below, the target trajectory is determined by combining the control trajectories and applying rate smoothing to them. To this end, i Associated with (or relative to) manipulating M i The trajectory of the i may be referred to as a “partial trajectory”, while a trajectory T that is associated with a target (or evaluated relative to a target) may be referred to as a “full trajectory”.

[0261] When considering targeting a specific target G i In the case of multiple trajectories, the symbol Τ can be used (n) Instead of Τ, it refers to the target G i The nth trajectory of .

[0262] The above simplifying assumption of a "one-to-one" relationship between partial trajectories and maneuvers has been found to give acceptable performance with a reduced computational burden. However, this does not make alternative implementations that implicitly assume a "one-to-one" relationship between trajectories and maneuvers feasible. Without this simplifying assumption, the more general Bayesian relationship still holds:

[0263]

[0264] Among them, p(Τ k |M j )∈[0,1] is a given manipulation M l Part of the trajectory j In this case, instead of assuming a mode / most likely partial trajectory, we can get the probability of j |M l) instead, partial trajectories are sampled, and all descriptions in this article about modes / most likely partial trajectories also apply to the sampled trajectories in this case.

[0265] Inverse Planning - Example Implementation

[0266] As a further illustration, an exemplary embodiment of inverse planning for target identification will now be described in further detail. This embodiment uses target-level inverse planning to predict the probability distribution P(G|O) of at least one external subject over the set available targets G, that is, for each available target, the estimated probability that the subject currently implements the target.

[0267] The described embodiments additionally use probabilistic maneuver detection to predict a distribution P(M|τ) over a set M of possible maneuvers given an observed trace τ of an external agent, i.e., for each possible maneuver, the estimated probability that the agent is currently performing that maneuver. Maneuver predictions feed into target predictions as described later. For example, maneuver detection can be implemented using inverse planning at the maneuver level.

[0268] In other words, the described embodiments reason at both the manipulative level and the target level.

[0269] Target Identification

[0270] Goal recognition is the process of inferring the subject's goal given observations of the subject's past actions and other contextual information.

[0271] In the present context, the goal will typically specify a target location for the vehicle, such as various exit points or different lanes on a highway / intersection, on which the inverse planner A24 reasons. Knowledge of another agent's goal can inform the planning process because predictions can be made about the agent's behavior for its assumed goal.

[0272] Using object recognition also increases the “explainability” of AV systems. Explainability refers to the ability of a system to explain its decisions to humans. Logging the object inference process (what the ego vehicle thinks the other vehicle’s object is at different times) can provide explainable information for decision tracking and debugging.

[0273] The target can be inferred deterministically or probabilistically. In the following example, the target is inferred probabilistically given a set of relevant observations O, that is, the target posterior P(G|O) is estimated for a finite set of available targets G, where P(G i |O) is the external subject with a target G given an observation value Oi The probability of ∈G.

[0274] In the context of MCTS, goals are inferred probabilistically in order to make inference predictions about the future trajectories of external agents when the tree is presented in the sense described above.

[0275] Control and Target

[0276] First, an example scheme for determining available goals and maneuvers is described. This description provides context related to a particular implementation of inverse planning described later. However, the description is not limited in this regard and is equally applicable to other implementations of MCTS, including alternative implementations of inverse planning / goal identification and implementations that do not use goal identification (e.g., the example given above of using a data-driven behavior model without higher-level goal identification).

[0277] Multi-strategy baseline

[0278] A "multi-strategy approach" is used as a baseline. The basic assumption of the multi-strategy approach is that all vehicles (or more generally, external actors) including the ego vehicle are performing one of a finite number of maneuvers at any time, such as following a lane, changing lanes, turning, etc. Such maneuvers may also be referred to as "maneuvering strategies" in this article (note the distinction between a maneuvering strategy (i.e., a maneuver of a finite set of maneuvers) and an action strategy that can be used to execute a selected maneuvering strategy; note also that the English / US spellings of maneuver / manipulation are used interchangeably in this article).

[0279] A "target" actor refers to an external actor whose behavior is being predicted. Predictions can be made for one or more target vehicles (or other actors), and the above set of assumptions is applied to each target actor. The manipulation strategy examples in the preceding paragraphs are actually "closed loops", which means that they take sensor feedback into account and automatically change their speed and the distance between the lead vehicle (the lead vehicle is the vehicle followed by the target vehicle). However, as explained below, for the purpose of inverse planning, they can be modeled as "open loop" manipulations, with the benefit of increased computational efficiency, and do not significantly affect the performance of the inverse planner A24. The target vehicle and the lead vehicle are cars in the following examples, but the description is equally applicable to any form of actor (vehicle, pedestrian, cyclist, etc.). Similarly, this example considers an autonomous vehicle (ego car), but the description is applicable to any form of autonomous vehicle.

[0280] The following is described in detail Fig.12 A manipulation detector 1102 is shown as part of the prediction component A4.

[0281] As an example, the maneuver detector 1102 may perform inverse planning at the maneuver level in the manner described above.

[0282] As another example, the maneuver detector 1102 may implement a Bayesian change point detection method. This method is used to segment the observed low-level trajectory of the target vehicle into a sequence of maneuvers, and the most recent segment is used as a prediction of the currently executed maneuver of the car. Using this prediction, multiple forward trajectories are simulated for each maneuver available to the ego car, and the maneuver with the best evaluation is chosen for execution.

[0283] Bayesian change point detection is known per se and refers to the probabilistic detection of changes in the underlying hidden state of a system. In the present context, the hidden state of an external actor at a particular time is defined as the manipulation strategy it is executing at that time. This hidden state is not directly observable and therefore needs to be inferred through observations caused by the hidden state (in this case, the observed traces). This allows estimating the probability of each available manipulation at the current point in time, i.e., the probability that the external actor is currently executing that manipulation.

[0284] For example, a hidden Markov model (HHM) can be used to represent manipulations as hidden states of the HHM, where the observed traces are modeled as being caused by state transitions of the HMM.

[0285] As described below, a relatively small set of maneuvering strategies can be defined to significantly simplify the planning process. Such a small set can be defined because the maneuvering strategies are defined in a way that contains low-level planning such as speed and distance. Similarly, since other cars are assumed to be executing one of these same strategies, their current maneuvers can be effectively inferred given the low-level trajectory.

[0286] This multi-strategy approach itself is short-sighted because it only predicts the current maneuvers of other cars, but it does not attempt to predict their future maneuvers. Inverse planning builds on this approach to predict future actions as described later.

[0287] Control

[0288] Basic Controls

[0289] Manipulations represent basic actions used for planning and prediction. In this example, consider the following "basic" (elementary) manipulators:

[0290] Follow Lane

[0291] Changing lanes left / right

[0292] Turn left / right (turn to the nearest lane in the driving direction)

[0293] Stop / Caution

[0294] It will be appreciated that the described techniques may be extended to alternative or additional manipulations.

[0295] Each elementary maneuver has a specified applicability and a termination condition associated with it. A maneuver is only available in a given state if the state satisfies the maneuver's applicability condition. For example, a lane change to the left is only possible if there is a lane to the left of the car and if there is enough open space for the car in that lane. The applicability conditions may also encode traffic rules. The maneuver terminates if the state satisfies the termination condition. For the lane change maneuver, this is the case once the car has reached the lane and is aligned with the lane direction.

[0296] The maneuver specifies a reference path for the ego-car to follow and a target rate along that path between the start and end of the maneuver. In the present context, a trajectory is defined as a pair (static reference path, target rate). That is, as a static reference path plus an associated set of target rates (see above).

[0297] Typically, however, a maneuver is associated with multiple trajectories. For example, in the context of prediction, a maneuver may be associated with a trajectory distribution, i.e., a set of trajectories that each have a specified probability (i.e., the probability that the external actor will follow the trajectory given that the external actor is currently performing the maneuver).

[0298] Some maneuvers such as lane following and caution do not have natural termination conditions. For such maneuvers, the termination conditions are specified as parameters. "Macro actions" (see below) automatically set these parameters based on context information.

[0299] The Caution maneuver is a variation of the Stop maneuver used in macro actions. Caution will cause the car to slow down and continue moving to a specified position given as a parameter. At that position, if the terminal condition is met, the maneuver terminates, otherwise it will stop the car completely and then terminate when the terminal condition is met. The terminal condition is used to check for oncoming traffic in a specified lane (given as a parameter). If no lane is specified, the terminal condition is always true. Allows the car to plan safe and smooth entries / exits when there may be oncoming traffic.

[0300] As a special case in the caution only for the left exit / right exit macro, a forced termination is allowed if the oncoming vehicle is stopped and predicted to remain stopped for at least the amount of time required for the controlled car to complete the turning maneuver. This allows special cases such as in scenario 3 described below.

[0301] Some basic maneuvers are only used inside macro actions, either because they have additional parameters or because they are only possible in very specific positions, such as a turning maneuver.

[0302] The following examples use two types of basic manipulation: open-loop manipulation for inverse planning (prediction) or more generally for target recognition, i.e., Figure 1 The target recognition component A24 in AV and the closed-loop control for MCTS self-planning in the AV planner A6 are described in detail in the following sections.

[0303] Closed-loop control

[0304] If feedback from sensors is used, the basic maneuvers are closed loop. This can cover different degrees of automation in the maneuvers. Here, it is assumed that each closed loop maneuver automatically controls the speed of the car and the distance to the leading vehicle (a form of adaptive cruise control (ACC)). It is also assumed that each maneuver can automatically initiate emergency braking.

[0305] The system is agnostic about the specific implementation of the controls and essentially treats them as "black box" functions. This has the benefit of flexibility and allows for straightforward incorporation of different representations. Possible implementations include:

[0306] Hard-coded heuristics using finite state machines or any programming logic

[0307] A lattice path planner using motion primitives defined by a dynamic bicycle model

[0308] Constrained Optimization Planner

[0309] After a state change, the planning methods in the above list can be called repeatedly to close the loop.

[0310] Open loop control

[0311] In contrast to closed-loop control, open-loop control does not use sensor feedback. Open-loop control specifies reference paths (or distributions over paths) without attempting to correct these paths with feedback. This allows for simpler and computationally less expensive implementation of the control.

[0312] Third, the system's representation of open-loop control is agnostic.

[0313] A simple but effective way to implement open loop control is to fit a polynomial function to a set of points extracted from the road topology. The target rate can be set to a constant or other time series. This is to calculate a trajectory rather than a distribution over trajectories, which has been found to be sufficient in many practical contexts.

[0314] Another embodiment uses a behavioral model of the type described above (such as a Gaussian process or a neural network) to collect driving trajectories. The trajectory data can be generated in different ways:

[0315] In simulation, use corresponding closed-loop control in different situations

[0316] From real driving data generated by AV platforms

[0317] From real CCTV data, segmented into identified manipulations. For example, Fig.13 An example of a Gaussian trajectory model 1202 fitted to trajectories observed over time in CCTV footage is shown.

[0318] Conditions such as "wait until upcoming traffic clears," as used in cautious maneuvers, may be implemented in open-loop maneuvers by waiting a specified time until the condition is predicted to be true (eg, wait until traffic is predicted to clear).

[0319] Target Rate and Rate Smoothing

[0320] As noted, in this example, a trajectory is defined as a static path combined with a set of target rates.

[0321] The target speed is set inside the base maneuver. As a general principle, it can be assumed, for example, that the vehicle will attempt to drive at a certain speed (e.g., the speed limit) in a certain area. This target is reduced if there is a slower car ahead (in which case the car's speed is the new target) or if required due to curvature in the driving path, etc.

[0322] For open-loop lane following, the target velocity is set to

[0323] min(speed limit, speed of the car ahead (if any)).

[0324] Due to its simplicity, this method is computationally inexpensive, but may result in conflicting predicted trajectories for other vehicles. For example, in Scenario 1 below, the truck is predicted to continue straight ahead at the speed limit (because there are no vehicles ahead), and car 2 is also assumed to overtake at the speed limit (during the lane following segment of the overtaking). Because the inverse planning for car 2 actually assumes a constant speed for the truck, the predicted trajectories for car 2 and the truck may conflict. However, in this and many other cases, it has been found that this conflict does not cause problems for the self-planner A6: safe and efficient planning decisions can still be made using the predicted output of this simple method.

[0325] In basic maneuvers that generate curved trajectories (e.g., turns and lane changes), a heuristic based on local curvature is used to set the target velocity. This function is as follows, where v target is the target rate, and ψ is the point curvature. The other terms are constants set to appropriate values

[0326] vtarget =max(v min ,v max -cψ)

[0327] The point curvature is given by the following equation, where x and y are Cartesian coordinates. The derivatives of x and y are estimated from the target path of the vehicle using finite differences:

[0328]

[0329] Rate Smoothing

[0330] Because the rate is set inside the base maneuver, two subsequent maneuvers can have a sudden change in rate. A typical example is lane following followed by a turning maneuver.

[0331] To compensate for this effect, the resulting complete trajectory across different maneuvers can be smoothed in rate to improve control and realism. A rate smoothing function is used that optimizes the target rate in a given trajectory. Rate smoothing is used for both prediction (inverse planning) and MCTS self-planning.

[0332] Rate smoothing is formulated as an optimization problem on a trajectory along a given path. Assume that the set of points at the longitudinal position and their corresponding target rates A continuous and differentiable function κ: x→v is fitted. Then, the time range T is considered H (which can be estimated using, for example, the target velocity), the time Δt that elapses between two time points, and other parameters of the optimization (e.g., the maximum speed v max and acceleration a max ), the smoothing problem is defined as:

[0333]

[0334] stx t+1 =x t +v t Δt

[0335] 0<v t <v max

[0336] |v t+1 -v t | max Δt

[0337]

[0338] Depending on the solution to the problem, spatial resampling can be used to obtain The actual achievable value at the position is given by x​N Given. If It is then possible to resample from this solution only.

[0339] Alternatively, one can start from x N Start solving similar problems and repeat the process until you reach the desired condition.

[0340] It is also possible to model other parameter(s) related to the trajectory, such as acceleration, and impose constraints on such parameter(s) as an extension to the problem.

[0341] Rate smoothing should take into account zero rates in the input trajectory, which indicates a complete stop. A simple way to achieve this is to split the trajectory into segments separated by stop events (zero rates) and apply a smoothing function to each segment.

[0342] Macro Actions

[0343] Special kinds of maneuvers, referred to herein as “macros”, ease the planner in two important ways: they specify common maneuver sequences, and they automatically set free parameters in basic maneuvers based on background information (usually the road layout).

[0344] In this example, the following macro actions are used (with manipulation parameters shown in brackets):

[0345] Continue Lane: specifies the sequence <Follow Lane (until visible lane end, i.e., lane target)>

[0346] Left / Right Exit: Specify the sequence <Follow Lane (until at / near turning point), Caution (until oncoming traffic is clear / safe distance), Turn Left / Right>. The exit point is automatically set to the exit point corresponding to the ego car's goal, allowing the macro action to skip earlier exit points (this is especially useful in roundabouts).

[0347] Crossing Roads: Specify the sequence <Follow Lane (until at / near road intersection), Caution (until oncoming traffic from the crossing road is smooth / safe distance), Follow Lane (until after road intersection)>

[0348] Overtaking: Specify the sequence <Lane change right, follow lane (until the ego car has passed other(s) cars), lane change left>

[0349] The applicability conditions of a macro action are given by the applicability conditions of the first maneuver in the macro action and possible additional conditions. For example, the additional condition for <right exit> is that the car is in the rightmost lane in its driving direction and the target exit point is in the same lane in front of the car. The additional condition for <crossing road> is that there is a road intersection, where the ego car must give way to the other car in front (see scenario 3 below). The termination condition of a macro action is given by the last maneuver in the macro action.

[0350] Macro actions can significantly speed up the planning process and the prediction process, because a single macro action may be sufficient to accomplish a given goal. For example, in the following scenario 2, if the goal of the ego car is G3, a single macro action <right exit> will have accomplished the goal. Macro actions are also useful because they automatically take care of transition points between maneuvers that would otherwise need to be explicitly planned (e.g., setting open termination conditions for maneuvers such as lane following).

[0351] The macro actions used in this work do not define a hierarchy of decomposable actions; they simply define sequences of actions in a flexible way. Macro actions are implicitly included in the planning search space.

[0352] As noted, for the particular exemplary MCTS process described below, all basic manipulations in macro actions are closed-loop, and for inverse planning, all basic manipulations are open-loop.

[0353] Goals and Goal Generation

[0354] Goals of the Ego Car

[0355] Based on route planning Ego Car The route planner ( Figure 1 A25 in the ego car planner takes as input an annotated road map (with lane directions, traffic signs, etc.) and a start / end position. It computes a route from the start to the end position that specifies the sequence of road segments and directions that the ego car must travel through. This route is passed down to the planner A6, where it is used to generate a goal for the ego car as follows:

[0356] Conceptually, planner A6 "zooms in" on the map, where a field of view (e.g., square or circle or any suitable shape) is centered on the ego car, similar to the scene picture shown below. This field of view moves continuously with the ego car. At any point in time, the goal of the ego car is given by the next exit point in the route (at which the car needs to change from the current road to a connecting road), or by the visible end of the current lane if the next exit point in the route is not yet visible. For example, in scenario 1 below, the next exit point is not visible, so the goal is G1. In scenario 2 below, the exit point becomes visible and the new goal becomes G3.

[0357] Other car targets

[0358] For example, the goal of the other car can be defined as its possible exit point and the visible end of the road in the direction of the car, bounded by the field of view of the ego car. A heuristic goal generation function is used to enumerate the set of possible goals for the car in a given field of view using information about the road layout and traffic rules (e.g., using an SDL road graph). To reduce computation time when there are many goals (e.g., many exit points), the function may ignore the exit point goal of a car if two earlier exit points already exist in the same lane ahead of the car.

[0359] Other types of goals can be specified, such as "I want you to be in front of me". Such goals can be added and removed dynamically based on the current configuration of cars in the environment. The following scenario description gives an example of a goal.

[0360] Target Identification

[0361] In this exemplary embodiment, target recognition is performed based on a “cost penalty”, as will now be seen. Fig.10 as described.

[0362] In the present context, "planning" means a sequence of one or more elementary maneuvers selected for reaching a goal. In some cases, such a sequence of elementary maneuvers may be defined by macro actions. With a macro action or some other sequence of multiple elementary maneuvers, each elementary maneuver is associated with a partial trajectory, and the partial trajectories are combined using rate smoothing to determine a complete trajectory for reaching the goal.

[0363] Fig.10 A schematic flow chart of a method for probabilistically inferring goals of external actors from a finite set of available goals based on cost penalties is shown. Fig.10 An illustrative example of the steps applied to a scenario with two available targets is shown on the right side of

[0364] 1. G1 continues along the current road, which is defined as the target position at the end of the visible road (more generally, defined as a reference point in front of the car on the current road);

[0365] 2. G2 takes a right turn to exit according to the exit location definition.

[0366] Given a set of possible goals for a car and a sequence of past elementary maneuvers performed by the car, the posterior distribution of the goals can be computed using a process of inverse planning. This method computes the posterior distribution of the goals given a sequence of observations O (e.g., the observation trace τ in the example above). n ), given the prior distribution P(G) on the target and the likelihood function L(O|G), calculate the Bayesian posterior P(G|O)~L(O|G)P(G) on the possible target G.

[0367] The notations P(O|G) and L(O|G) are equivalent and mean the conditional probability of observing the value O given the target G. The notation "L" is used consistent with the fact that these probabilities are likelihoods in a statistical sense (note that earlier in this disclosure the term likelihood was used in an everyday sense, and not necessarily this particular statistic. The meaning will be clear from the context).

[0368] The goal is defined in terms of the target location, and the symbol G i Can be used to indicate the target position of the zone. Target position G i It could be a point in space, but could also be a region or could correspond to a specific distance along a road, for example the target location could be defined as a line perpendicular to the road, and in this case the car is said to have reached the target once it reaches that line (regardless of its lateral position in the road).

[0369] Given a target G i ∈G likelihood L(O|G i ) is defined as the difference between the corresponding costs of two plans (cost penalty):

[0370] 1. From the initial position of the car t (at time t) to the target position G i The optimal plan is, that is, from r t To G i The optimal plan for r is obtained independently of any observed behavior of the car after time t. This can be performed as a sequence of elementary maneuvers, macro-maneuvers, or multiple elementary maneuvers in addition to macro-maneuvers. With multiple elementary maneuvers, the partial trajectories associated with them are combined to provide a plan for moving from an initial position r t Arrival at target G ithe optimal complete trajectory for t (regardless of any actual observed behavior of the car after time t); and

[0371] 2. The “best available” plan — defined as the plan from r given any observed behavior of the car between time t and time t+Δt t To the target position G i The optimal plan is, that is, from r t To G i The optimal plan is , where the additional constraint is that the plan must match the behavior actually observed in the subsequent time interval ΔT. In other words, as the car's initial position r t To target G i The optimal plan is calculated such that the plan complies with the observation O. This assumes that the car is more likely to execute the optimal plan to achieve the goal, but allows for some degree of deviation. This can also be performed as a sequence of elementary maneuvers, macromaneuvers, or multiple elementary maneuvers in addition to macromaneuvers. Through multiple elementary maneuvers, the partial trajectories associated with them are combined to provide a trajectory for moving from the initial position r t Arrival at target G i The "best available" full trajectory of , but taking into account the actual observed behavior of the car in the interval from t to t+Δt. The best available trajectory has an observation portion for the interval [t, t+Δt] that matches the actual observed trajectory, and a future portion for subsequent time intervals that is chosen so that the total cost associated with the best available full trajectory (i.e., the total cost of both the observation portion and the future portion) is minimized.

[0372] This is a form of goal recognition because the full path to the goal is considered (the full path may be based on multiple partial trajectories associated with multiple maneuvers).

[0373] The cost assigned to a complete trajectory can take into account various factors as described later. These include driving time (penalizing trajectories that take longer to reach the goal), safety (penalizing unsafe trajectories), and comfort (e.g., penalizing trajectories with excessive jerkiness).

[0374] For example, the initial position of the car is r t can be the first observed position of the car. A reasonable approach is to use a moving window of past observations defined by the sensor range of the ego car to define the initial position r t .

[0375] exist Fig.10In step 1004, an optimal plan is calculated for each goal G1, G2 (1 above). Once calculated, this allows the optimal trajectory to be determined for each goal G1, G2, for example using A* search (described below). After the optimal trajectory is calculated, the total cost associated with the optimal trajectory can then be calculated (also described below). The optimal trajectory is the complete trajectory, i.e., the trajectory for moving from the initial position r t to reach the goal in question.

[0376] exist Fig.10 In the example of , the trajectory is represented by points evenly spaced in time along the trajectory, so that evenly spaced points imply a constant rate, and an increase (respectively, a decrease) in the distance between points imply an acceleration (respectively, a deceleration). White circles are used to represent the optimal trajectory points. It can be seen that for the goal G1, the optimal trajectory is a straight path that continues along the road at a constant speed, while for the goal G2, the optimal trajectory gradually slows down as the car approaches the turning point for exit.

[0377] In step 1006, the best available plan is calculated for each target G1, G2 (2 above). As indicated, these consider the time t (when the car is at its initial position r t The actual observations O between time t) and the current time t+Δt. Those observations O may include observing the low-level trace τ, Fig.10 Indicated by black circles.

[0378] In this context, the observations O may alternatively or additionally include the current maneuvers of the car, ie the probability of each target may be estimated based on the maneuvers currently being performed by the car. They may additionally include maneuvers observed in the past.

[0379] Although not in Fig.10 , but as described above, probabilistic maneuver detection is applied to predict a probability distribution over the possible current maneuvers of the car. Therefore, the current maneuver may not be known explicitly but only probabilistically, according to the distribution p(M|τ) over the possible current maneuvers. This can be done by first sampling the current maneuver M from p(M|τ) j And then from this current control M j The target probability distribution p(G|O) is sampled (i.e., the observation O includes M j ) to solve it.

[0380] From the best available plan, the best available trajectory can be determined (described below), which in turn allows the full cost to be determined for the best available trajectory (also described below). t To the target position G i This is also a complete trajectory in the sense that the best available trajectory has the same Fig.10the observed portion between times t and t + Δt that matches (the black circles in) and further includes a future portion of the time after t + Δt, which is indicated by slanted - shaded circles in Fig.10 is shown in

[0381] In the depicted example, it can be seen that both the observed portion (black circles) and the future portion (slanted - shaded circles) of the best - available trajectory of target G1 match the optimal trajectory (white circles) of that target G1 quite well. Thus, the cost penalty for target G1, which is the difference between the cost of the optimal trajectory and the cost of the best - available trajectory, is relatively low.

[0382] However, for target G2, the observed trajectory (black circles) deviates quite significantly from the optimal trajectory (white circles) because the vehicle fails to reach the degree required by the optimal trajectory before time t + Δt. This difference itself will not necessarily incur a significant cost penalty (it can depend or not depend on the details of the implementation). However, due to the observed behavior, it can be seen that the future portion of the best - available trajectory (i.e., the portion after time t + Δt) must include hard braking (which reflects the fact that, given the context of the vehicle, the lowest - cost path from the vehicle's current position to G2 must involve hard braking), and this is penalized by the cost function. The difference from the cost of the optimal trajectory implies a higher cost penalty for target G2.

[0383] In step 1008, for each target G1, G2, the target likelihood L(O|G) is calculated based on the cost penalty, i.e., the difference between the cost of the optimal plan calculated in step 1004 and the cost of the best - available plan calculated in step 1006 for that target. This in turn allows the calculation of the target posterior P(G|O) based on the target likelihood and the target prior (step 1010).

[0384] The prior P(G) can be used to encode knowledge about the "intrinsic" probabilities of certain targets. For example, in Fig.10 the scenario, it can be observed that the vehicle relatively infrequently takes a right - turn exit, which can be encoded as a prior of P(G2) < P(G1). This will effectively bias target G1 towards G2. For scenarios without this prior knowledge, it can simply be assumed that, in the absence of any observations of the individual behavior of a particular vehicle, each target is equally likely, i.e., P(G1)=P(G2).

[0385] The above assumes that, given a target, the initial position r of the vehicle can be given tThe optimal plan for the goal can be determined given the observations in the subsequent time interval ΔT, and the best available plan for the goal can be determined given the observations in the subsequent time interval ΔT. Furthermore, it is assumed that given the optimal (respectively best available) plan, the optimal (respectively best available) trajectory can be determined. As will now be described, one mechanism for mapping goals to plans to trajectories in this manner uses A* search.

[0386] Probabilistic Programming Identification Algorithm

[0387] A* Search

[0388] Perform an "A* search" on open-loop maneuvers including macro actions. Filter maneuvers based on their suitability criteria. If the basic maneuvers specify a distribution over trajectories, a suitable single trajectory can be selected, for example, as a model or most likely trajectory.

[0389] A* search is a known method based on a weighted graph of nodes and edges. In the present context, its goal is to find a way to reach a specified goal G from a given location (represented by a start node). i The optimal plan for a problem is to find a sequence of nodes (and thus a sequence of actions) that reaches a goal with relatively low cost. For each node n, the "cost so far" g(n) is defined as the cost from the start node to node n, and the heuristic function h(n) provides an estimate of the cost from n to the goal. The search starts at the start node, and at each node, the sequence is extended to the next node with the lowest node cost, which is defined as:

[0390] f(n)=g(n)+h(n).

[0391] A simplifying assumption can be used: all other cars in the environment use a constant velocity model.

[0392] In order to find the initial position r t To target G i The optimal trajectory is searched from time t and position r t The maneuver performed by the car begins. In order to find the current position r of the car given the observation value O in the time interval Δt t+Δt The best available trajectory to the target, searching for the current maneuver detected and the current position r t+Δt start.

[0393] The above-described maneuver detection method can be used to detect maneuvers. Note that this will generally result in a probability distribution over possible current maneuvers. Inverse planning should be performed for each predicted current maneuver, since these maneuvers may result in different posteriors on the target. Therefore, each possible current maneuver posteriorly produces its own target. For computational efficiency, only the n most likely current maneuvers are considered, and any other maneuvers are ignored, where n is a method parameter that can be set appropriately.

[0394] The cost g(n) to reach the current location (current search node) so far can be defined as the estimated driving time to the current location based on the path and speed specified by the basic maneuvers.

[0395] The cost heuristic h(n) that estimates the remaining cost to the goal is given by the travel time from the current location to the goal location via a straight line at the speed limit. Note that in order to guide the search effectively, h(n) only needs to provide an estimate of the minimum remaining cost in order to guide the search. Therefore, relatively rough assumptions can be made here without affecting performance.

[0396] After finding the optimal plan, a complete trajectory is generated across the maneuvers in the plan. Then, rate smoothing is applied to the complete trajectory, and the full cost function is calculated (see the cost function section below) to obtain the final cost of the plan. This final step (applying smoothing and the full cost function) makes it possible to pick up additional "clues" based on rate, such as in the braking example below.

[0397] Rate smoothing is not applied to any part of the observed trajectory. Otherwise, the effect of rate smoothing could be to remove evidence that would suggest a target at some point (see Fig.10 ) for further details in the context of the examples.

[0398] In order to consider the situation where the car is in the middle of a maneuver (eg, changing lanes), the maneuver currently being performed by the car needs to be identified so that the maneuver currently being performed can be completed before planning further into the future.

[0399] To reduce the search space, basic turning maneuvers and cautious maneuvers can be removed from the search space since these are not used outside of macromaneuvers. If the terminal condition of a lane following maneuver is set to a certain fixed length as a default value, the lane following maneuver can still be used as a basic maneuver outside of macromaneuvers.

[0400] This example does not check for conflicts between external actors during inverse planning. Although it is possible to do so, it will make the overall process more expensive. In addition, due to some simplifying assumptions such as the constant speed of other vehicles, there may be situations where conflicts are inevitable. The multi-agent MCTS method detailed below can handle such conflict checks.

[0401] Alternatively, the process can be extended to reason about possible conflicts as part of inverse planning.

[0402] It has been found that the assumptions set forth above with respect to A* search provide a good balance between computational efficiency and performance. Of course, it will be appreciated that a different set of assumptions may be used, such as a non-constant rate model. As another example, the above essentially transforms the search problem into a deterministic problem by considering only the most likely trajectory associated with the maneuver. However, this can be equally extended to probabilistic searches based on full or truncated trajectory distributions. That is, while the above considers full / partial trajectory models in the form of a single trajectory, these techniques can also be applied to full / partial trajectory distributions.

[0403] Multithreading can be used to speed up the process by using a separate thread for each target G. Under the additional assumption that the cars have separate targets, further parallelization can be achieved by using one process / thread for each other car.

[0404] brake

[0405] Although braking is not identified as a unique maneuver in the above method, this method can still account for braking in the likelihood of the target.

[0406] As an example, consider Fig.10 The scenario shown in and described above. The optimal plan from the car's initial position to G2 is given by the right exit macro action. After applying rate smoothing, the result is a trajectory that continues to decelerate until the car reaches the turning point. The cost of this optimal trajectory is denoted by C2.

[0407] Suppose the car is observed following the lane, now closer to the halfway point of the turn, and decelerating at a similar rate to the optimal trajectory of G2. So, if the trajectory to G2 had competition, it would result in a trajectory with a cost similar to that of C2. On the other hand, the optimal trajectory from the initial state to G1 would not involve such braking, resulting in a significant cost difference (due to the time difference and possible jerkiness). Therefore, the posterior probability of G2 increases, while the probability of G1 decreases.

[0408] (As an aside: note that rate smoothing is not applied to any part of the trajectory that has already been observed.) In this example, the reasoning behind this is obvious: if rate smoothing is applied to the entire trajectory, then smoothing will also reduce the rate in the part that has already been observed following the lane maneuver. This will result in a lower cost penalty because the braking is no longer as abrupt. In other words, it will make the trajectory more similar to the optimal trajectory from the initial observation position. Therefore, smoothing is suitable for treating the past trajectory as an unmodifiable constant.)

[0409] Now suppose that the car is observed to follow the lane but with a constant velocity instead of decelerating. If the trajectory continues to G2, it will involve sudden braking, resulting in a much higher cost than C2. On the other hand, continuing the trajectory to the goal G1 (staying in the lane) is essentially the optimal plan from the initial state to G1. Therefore, in the posterior probability G2 decreases and G1 increases.

[0410] Finally, suppose the car is observed following the lane and suddenly slows down approaching the turning point. This sudden braking will result in a significant cost penalty for both G2 and G1 since it is not in the optimal trajectory from the initial state. Therefore, the costs of both goals will increase, resulting in a smaller change in the relative probabilities of the two goals.

[0411] Target identification via target tree

[0412] As an alternative to inverse planning methods, decision trees can be used to achieve object detection.

[0413] Fig.11 An example of a decision tree for object recognition is shown in .

[0414] Given a target set G, using Bayes' theorem, the posterior distribution P(G|O)~L(O|G)P(G) on the target is calculated. The likelihood L(O|G) of the target G is calculated by the tree associated with G, which can be generated jointly with G. The tree calculates a normalized score in the range [0, 1] to indicate the likelihood of the target, with 0 indicating no support for the target.

[0415] An example tree for goal G2 in scenario 2 (right exit) using example numbers is shown on the right. The tree tests several conditions, including:

[0416] Is the car in the correct lane for the target?

[0417] Is the car indicating the correct direction?

[0418] Whether the exit point is next in sequence for the car (i.e., there are no other exit points closer to the car in the same lane)

[0419] Whether the car is slowing down

[0420] In each branch, the tree tests for a condition and multiplies the score (starting with 1) by some factor in the range [0, 1], resulting in a final score at the tree's terminal node. Different branches may test for different conditions, and the factors given in the branches may be different to reflect the different importance of the conditions tested. An alternative to multiplying factors is to add factors along the tree branches and normalize by the terminal node depth.

[0421] This approach of using trees has two main advantages: trees are cheap to evaluate, and trees are easily interpretable by humans (increasing interpretability).

[0422] Decision trees can be built based on real-world or simulated driving behavior data (e.g., from simulation or CCTV) using algorithms such as C4.5 or algorithms for regression tree learning. For example, a set of binary features (test conditions) can be specified, and the tree learning algorithm then finds the optimal subset of features and part scores given the ground truth data recorded for a particular target.

[0423] Predict trajectory based on target

[0424] To predict possible trajectories for the car to a given goal and the associated probabilities, the same A* search method as for inverse planning can be used. Instead of terminating A* after finding the optimal plan, a fixed time budget is imposed and the algorithm is allowed to compute a set of plans with associated costs (perhaps up to some fixed number of plans). Each time the A* search finds a node that reaches the goal, the corresponding plan is added to the set of plans. Trajectories can be generated from plans by connecting partial trajectories given by maneuvers in the plan.

[0425] The distribution over trajectories extracted from the plan can be computed using a softmax distribution (trajectory model):

[0426]

[0427] Among them, (n) is the nth complete trajectory to the target, C n It is the trajectory (n) The cost (the total cost after smoothing), and β n It is the trajectory (n) The scaling factor (as mentioned above, symbol T (n) The nth complete trajectory to the target is used to match the given control M j Part of the trajectory j The scaling factor can be used to down-weight specific trajectories based on observed data, i.e., to allow the trajectory prediction model to be trained on relevant examples. Alternatively, a constant scaling factor (e.g., a value of 1) can be used.

[0428] The above softmax trace model can be used as an alternative to the trajectory model obtained via RRT.

[0429] As in inverse planning, such a set of trajectories is generated for each predicted current maneuver, focusing on some subset of the current maneuvers such as the n most likely ones. A* starts after completing the current maneuver, and the final trajectory includes the remainder of the current maneuver.

[0430] This predicts a variety of plausible trajectories rather than a single optimal trajectory, which is beneficial since there are situations where multiple trajectories achieve (close to) minimum cost but result in different predictions, which in turn may require different behaviors on the ego vehicle part.

[0431] Integrating Object Recognition with MCTS

[0432] We will now describe the specific mechanisms used to integrate target recognition with the MCTS manipulation planning process described above. This combination provides a general purpose interaction-aware manipulation planner.

[0433] Sample current control and target

[0434] Fig.12 shows a schematic overview of the mechanism by which the results of inverse planning can be applied as part of an MCTS demonstration. Fig.15 The described principles are applied where MCTS is performed multiple times with different sampling of probability predictions about (one or more) other agents.

[0435] The manipulation detector 1102 detects the current manipulation probability distribution of the target actor in the above manner. i ∈M (M is a finite set of available manipulations), which provides the probability that the target actor is currently implementing the manipulation given the observed low-level trace τ, i.e.

[0436] P(M|τ).

[0437] Each MCTS super iteration starts as follows. For every other car:

[0438] 1. Sample the current control from the predicted current control and its probability set, that is, based on P(M|τ) for the current control M j Take a sample ( Fig.12 1104);

[0439] 2. Sample the target from the target posterior P(G|O) corresponding to the sampled current control, i.e., apply the above with the sampled current control M j (which can be considered to form part of the observation value O).

[0440] These samples, namely the samples of the current control M j and the sampling target G iIt is used throughout the super iteration of the MCTS process. Steps 1 and 2 above will be performed at the beginning of each MCTS super iteration, where iterations are applied at each node of the tree (from the root node to the terminal node).

[0441] exist Fig.12 In the shorthand notation P(G|M i ) and P(G|M k ) are used to represent the given current control M i and M j , note that the observation O can be broader than just the current manipulation (e.g., including a sequence of previous (one or more) manipulations). Using this notation, if we i Sampling is performed, then the target posterior P(G|M i ) for sampling; and if M j is sampled, then the target posterior P(G|M k ) will be used.

[0442] Sampling from the combined posterior prevents inconsistent presentation. For example, consider the case where the car has two possible goals and two predicted current maneuvers of specific equal probability. The goal posterior for the first maneuver places all probabilities on the first goal, while the goal posterior for the second maneuver places all probabilities on the second goal. Therefore, the combined goal posterior places equal probabilities on both goals (in contrast, if one were to sample a goal from the combined goal posterior and then sample the current maneuver, one could end up with a current maneuver for which there is no trajectory to the sampled goal (e.g., there is no path from the first maneuver to the second goal)).

[0443] Single-agent MCTS with trajectory sampling

[0444] Given the sampled current maneuvers and goals of each other car, their motion trajectories are sampled from the associated predicted trajectories and probability sets. These trajectories are used to predict the motion of each car along the display.

[0445] Self-planning is done on a set of closed-loop maneuvers, including macro-actions, that are applicable in the current state. In many cases, a single macro-action may be sufficient to achieve the goal of the ego car. The goal is to find such a macro-action as quickly as possible during planning. A simple approach is to sort the set of maneuvers so that macro-actions are always tried first; alternatively, a "best first guess" function can be provided, which, for a given goal, returns a macro-action that typically achieves the goal. If a macro-action is not currently applicable (e.g., exiting left when the ego car is in the middle lane), the planner should ideally quickly find a way to make it applicable (e.g., first using a basic left lane change maneuver, then using a macro-action to exit left).

[0446] After a maneuver is chosen for the ego vehicle, the environment is simulated forward until the end of the maneuver. This includes simulating the trajectories of the other cars forward to the same point in time as the ego vehicle, as well as any other modeling processes in the environment. During the forward simulation, a conflict checker is applied to check whether the ego vehicle is in conflict with any other vehicle. If there is a conflict, that branch in the search tree is immediately "cut off" (i.e., not explored any more).

[0447] MCTS uses a cost function defined in the following sections. To ensure that MCTS only generates plans that achieve the goal, the cost function is applied only if and when the presentation reaches the goal. More specifically, the cost function is applied to the trajectories generated by the maneuvers in the presentation, after applying rate smoothing. The cost of the resulting smoothed trajectory is then propagated up the search tree. Thus, MCTS searches only for feasible plans that achieve the goal, and among these plans, searches for the best plan.

[0448] The use of macro actions means that a successful plan will typically contain 1-5 maneuvers (including macro actions). So, to speed up the search, the MCTS search can be cut off (terminated) at depth 5.

[0449] A pseudocode algorithm for single-agent MCTS with target recognition is provided in Appendix B.

[0450] Speed ​​Enhancement

[0451] Possible speed enhancements include:

[0452] Various degrees of multithreading can be used in MCTS, for example, using multiple threads to generate displays and update statistics.

[0453] By planning open-loop control instead of closed-loop control at the expense of reduced planning quality,

[0454] The computational cost can be reduced. The manipulation to be performed after MCTS planning is always closed-loop.

[0455] Cost Function

[0456] Cost functions are used in inverse planning and MCTS to quantify the cost of a driving trajectory. A trajectory T contains a reference path for the vehicle to follow and a sequence of target rates along that path. The cost function takes a trajectory as input and returns a real-valued cost C(T). For any given trajectory, the evaluation of the cost function should be cheap, and the computational cost should grow by at most O(|T|).

[0457] For example, cost can be defined as a weighted (linear) sum of several cost factors. These factors include:

[0458] Time: How long does it take the car to complete the trajectory? (Note: this is different from the length of the reference path, as it also depends on the target rate.)

[0459] Curvature: How "curvy" is the trajectory? Tightening the curvature will result in exponentially increasing cost. Angular rate can be used as a proxy.

[0460] Rate change: How much does the rate change along the trajectory? High rates of change (e.g., sudden deceleration) should result in exponentially increasing costs.

[0461] “Safety”: As a proxy for safety, a “cost factor” can be calculated with an exponential cost based on the minimum distance to any other vehicle during the trajectory (ignoring vehicles that exceed the distance threshold).

[0462] That is, the cost increases exponentially as the minimum distance decreases.

[0463] Most of these cost factors can be calculated iteratively by stepping forward in time through the trajectory and doing some simple calculations based on the old / new position vectors and the target rate. The weighting of the cost factors will need to be adjusted based on empirical testing.

[0464] Normalization

[0465] To ensure comparability of different cost factors, it is helpful to normalize all cost factors to the same range [0, 1]. It also helps to normalize the weights so that the weights are non-negative and add up to 1. Together, these normalization steps ensure that C(T) is in [0, 1], which satisfies the assumption in UCB / UCT that costs are in the range [0, 1].

[0466] Normalization may be easier to implement if the cost factor is defined as a min / max function (e.g., maximum yaw rate, minimum distance to other cars).

[0467] An alternative and cheaper approach is to associate costs with maneuvers rather than with trajectories. For example, define the planning cost as the number of elementary maneuvers in the plan. This can be an appropriate approach if the trajectories generated by maneuvers (e.g., lane changes that always follow the same variety of strips) do not vary or vary only slightly. However, if maneuvers generate bundles of trajectories, or if the trajectories can depend on dynamic aspects such as vehicle configuration (e.g., lane changes close behind the vehicle vs. lane changes not so close behind the vehicle), then assigning a fixed cost to maneuvers may not be appropriate, because the same maneuver can result in significantly different trajectories depending on the context.

[0468] Example scenarios

[0469] Three exemplary scenarios will now be described to illustrate the concept of a target and to highlight how target recognition can inform the ego car’s decision making. These scenarios also illustrate some edge cases of the proposed approach. FIG. 14A to FIG. 14C Three scenarios are shown in Figure 2.

[0470] Scenario 1: Car overtaking a slow truck ( Fig.14A )

[0471] Car 2 is driving behind a slow-moving Truck 1. A target position for Car 2, denoted as G1, is shown. In this scenario, Car 2 may move to the ego lane to overtake the truck.

[0472] The myopic baseline alone would place most / all of the posterior probability in the following lane maneuver of car 2, failing to predict that a lane change might be imminent because it does not reason about future maneuvers.

[0473] Assuming that car 2 uses a similar cost function as the ego car (i.e., including a time factor), and that inverse planning is performed for car 2 to achieve G1, this method can predict a lane change maneuver for car 2 following through the following lane at the speed limit. If car 2 is far away from the ego car, this prediction will likely not result in a reaction from the ego car. However, if car 2 is close to the ego car (as in the picture), the ego car may decide to slow down and keep a certain distance from car 2.

[0474] Scenario 2: Car changes lanes to exit ( Fig. 14B )

[0475] The two possible targets G1 and G2 for the other cars are generated based on the road layout. For car 2, both G1 and G2 are possible, but for car 3, only G1 is possible.

[0476] Car 2 is observed to be performing a right lane change. Based on the inverse plan for car 2 and every possible goal, the lane change most likely indicates G2. After the lane change, the optimal plan for G2 dictates following the lane, reducing speed (due to rate smoothing) and then turning right. Given this sequence and assuming the ego vehicle's goal is G1, the MCTS planner can realize that car 2's deceleration over-penalizes the ego vehicle objective function (because it forces the ego vehicle to slow down as well), and therefore, after car 2 has completed the lane change, it can decide to change lanes to the left.

[0477] The nearsighted baseline will only recognize that car 2 is changing to the right lane, but will not be able to predict the subsequent deceleration. The baseline will likely still perform a lane change, but only after the deceleration is already in progress.

[0478] Scenario 3: While waiting for the car ahead to move ( Fig. 14C )

[0479] Indicates four possible destinations for other cars based on the road layout. The ego car is waiting at the intersection.

[0480] No deceleration is observed for car 3, indicating that G2 is most likely (but G3 / 4 are still possible, albeit at the expense of heavy braking).

[0481] Car 2 has stopped at the position shown, matching the plan to reach G3: because Car 3 is approaching and using a constant velocity, Car 2 must stop, wait for Car 3 to pass, and then turn right to G3. In contrast, stopping at the position shown does not indicate G1 or G4 (G2 is not a valid destination for Car 2).

[0482] Given the inferred goal and planned clusters, the ego car can realize that it can safely turn left onto the road while Car 2 is waiting for Car 3 to pass.

[0483] The myopic baseline will identify the stopping maneuver of car 2, but will not be able to predict subsequent maneuvers.

[0484] Data-driven behavioral models

[0485] As described above, the learned agent behaviors can be used within an MCTS simulation to provide a realistic simulation. Such generative behavior models can be used in conjunction with, or as an alternative to, inverse planning, i.e., the learned generative behavior models can be used without performing inverse planning (e.g., the learned model can be used to simply generate one or more trajectories without reasoning about goals, maneuvers, etc.). As noted, examples of such models include spatial Markov models and neural networks.

[0486] For example, the generated behavior model can be a spatial Markov model (SMM) based on multiple spatial zones within the monitored driving area, wherein at least one of the following is calculated: an estimated occupancy probability associated with each spatial zone, and an estimated transition probability associated with each spatial zone in a plurality of spatial zone pairs.

[0487] A spatial region may be a cell of a grid overlaid on the monitored driving area. This may take into account road structures and / or other structures in the monitored driving area, which may be manually annotated or determined, for example, from a map.

[0488] Symbol p i represents the estimated occupancy probability of spatial region i, and p i,j represents the estimated probability of transition from spatial region i to spatial region j. With SMM, given a model Model, p(Τ|Model) can be determined based on the occupancy probabilities and / or transition probabilities associated with a series of grid cells (or other spatial regions) traversed by a driving path (trajectory) Τ.

[0489] The driving behavior data may include any form of sensor data, such as image data and / or motion sensor data, etc. The data may be collected in any suitable manner, but CCTV (closed-circuit television) systems provide a particularly convenient means of collecting driving behavior data, particularly in urban environments with good CCTV coverage. For example, the present disclosure recognizes that CCTV from complex driving contexts (e.g., complex roundabouts, multi-lane intersections, blind corners, etc.) provides a rich source of driving behavior data.

[0490] In the following examples, real life driving behavior data (such as CCTV image data) is used to generate a model for training and for predicting the behavior of an agent while driving.

[0491] Real life data can be collected for a fixed area over a period of time. The time period for collecting data can be, for example, 24 hours, in an attempt to generate an accurate representation of the movement of traffic through the area. The location can be small, such as an intersection. Areas with high traffic volumes passing through them can be selected to maximize the likelihood of data collection.

[0492] Collect data about the road layout (driving context). This can come from maps such as HD (High Definition) maps, or it can be collected from CCTV footage and entered manually or automatically. For example, CCTV footage can be manually annotated.

[0493] Information about the positions and movements of the actors in the collected data is extracted from the collected data and used to build a spatial Markov (state transition) model (SMM) of normal driving behavior. The state transition model is a discrete unit approximation model that can be used to provide a simplified representation of normal behavior. To achieve this, a grid can be applied to the locations captured in the CCTV footage. The grid can be in the range of 5-10 cm / unit.

[0494] The behavior of the subjects in each grid cell can then be analyzed over the time period in question. The information extracted in this analysis can, for example, include the frequency of occupancy of each grid element over the time period for which the data was collected and the number of transitions made from each element to its surrounding elements during that time period. This information can then be used to estimate the occupancy probability p i Assign to each grid cell and estimate the transition probability p i,j Assigned to each pair of grid cells i, j.

[0495] For example, Fig.16 The SMM determined for a road layout within the field of view of a CCTV camera is shown. In this case, the road layout is part of a relatively complex intersection on which a fixed grid has been superimposed. By way of example, two cells of the grid are denoted by reference numerals i and j and have occupancy probabilities p determined by observing real-world traffic. i and p j The transition probability p between units i and j is i,j is the probability that the agent moves directly from cell i to cell j, also determined through observation.

[0496] As will be appreciated, the above description considers only some examples of suitable models for modeling expected vehicle behavior. Other examples of suitable models include Markov decision process models and rewards for data. In this case, training is performed by fitting a Markov decision process model and rewarding data.

[0497] At the hardware level, the AV's onboard computer system A1 includes execution hardware capable of executing algorithms to perform the above functions. Although the execution hardware can be general or special execution hardware or any combination thereof, it will generally include one or more processors (such as a central processing unit (CPU)) and it can be combined with special hardware operations, such as but not limited to accelerators (e.g., GPUs), field programmable gate arrays (FPGAs) or other programmable hardware and / or application-specific integrated circuits (ASICs), etc. Given the need to perform complex data processing operations, often using sophisticated and complex ML / AI models, with sufficient accuracy and speed (often real-time) to ensure safe and reliable operation, the onboard computer system may be highly sophisticated, possibly with specialized computer hardware that is customized to implement the models and algorithms in question. In particular, given the speed at which innovation is progressing in the field of AI, it will be appreciated that the architecture of the AV onboard computer system A1 at both the hardware level and the functional / software level can take many forms. Herein, functional components implemented in a computer system, such as data processing component A2, prediction component A4, and AV planner A6, are high-level representations of specific functions implemented by the computer system, i.e., functions performed by (a combination of) any general and / or special-purpose hardware of the computer system that is appropriate in the context.

[0498] Appendix A - Example of scenario description

[0499]

[0500]

[0501]

[0502]

[0503]

[0504] Annex B

[0505] Pseudocode for single-agent MCTS with target recognition

[0506] When calculating the budget not fully spent

[0507] For each of the other cars, sample the current maneuver, goal, and trajectory and perform a display from the current state until the goal is reached or the maximum display length is reached:

[0508] Select self-manipulation from the current state / / For example, use UCB rules or some other heuristics, along with a priority mechanism for macro actions, to simulate self-manipulation until it terminates / / If the manipulation is a macro action, we simulate through the entire sequence within the macro action

[0509] Using the trajectories they sample, simulate all other cars moving forward to the same point in time as the ego car If the ego car collides with another car during a simulation segment: cut off the search branch and choose a different maneuver

[0510] If the new state reaches the goal: extract the ego trajectory, apply rate smoothing, and compute the cost function propagation resulting in an increase in the search tree cost

[0511] Choose the best action for the current state (e.g., minimum expected cost or most visits)

Claims

1. A method for planning an autonomous driving vehicle AV, comprising: In the AV planner, a tree search algorithm is executed to determine a sequence of AV maneuvers corresponding to a path through the constructed game tree; wherein the game tree has a plurality of nodes representing expected states of encountered driving scenarios, and an expected driving scenario state of each child node is determined by updating the driving scenario state of a parent node of the child node based on (i) a candidate AV maneuver and (ii) an expected behavior of at least one external agent in the encountered driving scenario; and Characterized in that the expected behavior of the external subject is simulated by applying an inverse planning method to the observed traces of the external subject within a time interval, the inverse planning method comprising: determining a set of available goals or maneuvers for the external agent (502) in the encountered driving scenario, and for each of the available goals or maneuvers, determining an expected trajectory model within the time interval, comparing the observed trace of the external agent (502) to the expected trajectory model for each of the available targets or maneuvers to determine the likelihood of the target or maneuver, and The determined likelihood of at least one of the goal or the manipulation is used to simulate expected behavior of the external agent (502).

2. The method according to claim 1, comprising: An AV control signal for executing the determined AV manipulation sequence is generated.

3. The method according to claim 1 or 2, comprising: receiving sensor inputs related to the AV; as well as The sensor inputs are processed to determine an encountered driving scenario.

4. The method according to claim 3, wherein: Object tracking is applied to the sensor input to track at least one external subject in an encountered driving scene and thereby determine an observed track of the external subject over a time interval.

5. The method according to claim 1, 2 or 4, wherein: The step of simulating the expected behavior of the external agent using the determined likelihood of at least one target or maneuver includes: calculating at least one predicted trajectory of the external agent based on the expected trajectory model of the at least one target or maneuver and the determined likelihood of the target or maneuver.

6. The method according to claim 1, 2 or 4, wherein: The expected trajectory model for each target or maneuver is the distribution of predicted trajectories associated with that target.

7. The method according to claim 6, wherein: Each target G i The distribution of includes the conditional probability p(Τ|G i ), and the likelihood of the target p(G i |τ) is used to estimate at least one predicted trajectory probability p(Τ|τ) given an observed trace τ.

8. The method according to claim 1, 2, 4 or 7, wherein: The observed traces are used to predict a best available trajectory model for the target or maneuver, and the comparing includes comparing the best available trajectory model to the expected trajectory model.

9. The method according to claim 8, wherein: The likelihood of the at least one target is determined, the observed trace is used to predict a current maneuver and / or a future maneuver of the external subject, and the predicted current maneuver or future maneuver is used to determine the best available trajectory model.

10. The method according to claim 9, wherein: A sequence of a plurality of maneuvers is determined for at least one objective, and the best available trajectory model is determined for the objective based on partial trajectory models respectively associated with the plurality of maneuvers.

11. The method according to claim 10, wherein: Each partial trajectory model comprises one or more target motion values, and one or more motion values ​​for a future portion of the best available trajectory model are determined by applying motion smoothing to the target motion values.

12. The method according to any one of claims 4, 9 to 11, wherein: The expected trajectory model for each target is a single expected trajectory for that target.

13. The method according to claim 10 or 11, wherein: The best available trajectory model for each target is a single best available trajectory.

14. The method according to claim 13, wherein: The partial trajectory model for each maneuver is the most likely partial trajectory for that maneuver.

15. The method according to claim 8, wherein: A defined cost function is applied to both the expected trajectory model and the best available trajectory model for each objective or maneuver to determine respective costs for the expected trajectory model and the best available trajectory model, wherein the comparing includes comparing the cost of the expected trajectory model to the cost of the best available trajectory model.

16. The method according to claim 15, wherein: The cost function rewards reduced driving time while penalizing unsafe trajectories.

17. The method according to claim 16, wherein: The cost function also penalizes lack of comfort.

18. The method according to any one of claims 1, 2, 4, 7, 9-11 and 14-17, wherein: The expected behavior of the external agent is simulated by sampling targets from a set of available targets or maneuvers based on the determined likelihood of the target or maneuver.

19. The method according to claim 18, wherein: Determine the likelihood of at least one target, determine different target distributions for different maneuvers or maneuver sequences; and simulate the expected behavior of the external subject by sampling maneuvers or maneuver sequences from a set of possible maneuvers based on the maneuver distributions, and then sampling the targets from the set of available targets based on the target distributions determined for the maneuvers or maneuver sequences.

20. The method according to claim 19, wherein: The control distribution is determined using inverse planning at the control level, including: For each of the possible maneuvers, determine the expected trajectory model, and The observed trace of the external agent is compared to the expected trajectory model for each of the possible maneuvers to determine the likelihood of the maneuver.

21. The method according to claim 18, wherein: The tree search algorithm is executed multiple times on a plurality of constructed game trees with different sampling targets, and an AV manipulation sequence is determined by the AV planner based on a statistical analysis of paths within the plurality of constructed game trees.

22. The method according to claim 3, wherein: applying object tracking to the sensor input to track at least one external subject in an encountered driving scene and thereby determine an observed track of the external subject over a time interval; Wherein, the reverse planning method comprises: determining a set of possible maneuvers for the external subject in an encountered driving scenario, For each of the possible maneuvers, determining an expected trajectory model, comparing the observed trace of the external agent to the expected trajectory model for each of the possible maneuvers to determine the likelihood of the maneuver, and The determined likelihood of at least one of the maneuvers is used to simulate expected behavior of the external agent.

23. The method according to claim 22, wherein: The expected trajectory model for each maneuver is a single trajectory associated with that maneuver.

24. The method according to claim 22 or 23, wherein: The expected trajectory model for each maneuver is a distribution of predicted trajectories associated with that maneuver.

25. A computer system comprising execution hardware configured to perform the method steps of any preceding claim.

26. A computer program comprising executable instructions configured to implement the method of any one of claims 1 to 24 when executed.

27. An autonomous vehicle (AV) planner configured to implement the method of any one of claims 1 to 24.

28. An autonomous vehicle comprising an autonomous vehicle planner according to claim 27 and a drive mechanism, wherein the drive mechanism is coupled to the autonomous vehicle planner and responds to a control signal generated by the AV planner.