Monte Carlo Policy Tree Decision Making

Abstract

An uncertainty-aware framework for highly variance planning problems with multiple dynamic agents is presented. Planning when surrounded by multiple uncertain dynamic agents is challenging as we cannot be sure of the initial states or future actions of these agents, leading to an exponential explosion in possible futures. Many important real-world problems such as autonomous driving fit this model. To address these challenges, multi-policy decision-making MPDM is combined with Monte Carlo tree search MCTS, where the policy tree search is implemented with marginal action-cost estimation and iterated belief particles.

Description

[Technical field]

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 264977, filed December 6, 2021, and also claims priority to U.S. Utility Application No. 18 / 074944, filed December 5, 2022. The entire disclosures of the above applications are incorporated herein by reference.

[0002] Government Provisions This invention was made with Government support under 1830615 awarded by the National Science Foundation. The Government has certain rights in this invention.

[0003] The present disclosure generally relates to a tree decision-making framework for problems where the marginal cost of each action is available and important, such as autonomous vehicle planning. [Background technology]

[0004] Planning with uncertainty is difficult because the compounding of uncertainty and the ruling out of each source of uncertainty is exponential in the number of possibilities. First, the space of possible action sequences grows exponentially with the length of the planning horizon and the number of dynamic agents. Second, uncertainty about the future world necessarily increases the further into the future one plans. The first difficulty poses a computational challenge, while the second means that continuous re-planning is necessary to take advantage of new information.

[0005] Planning under uncertainty is often modeled as a Partially Observable Markov Decision Process (POMDP), with a discrete set of states, actions, and observations, and with probabilistic state transitions, observations, and reward functions. Solving real-world POMDPs exactly is intractable due to the exponential nature of the probabilistic belief space. Tools that nearly solve exact POMDPs are only still tractable for small discrete problems. More realistically, this computational cost can be made tractable through the use of heuristics, sampling approaches, or domain-specific modeling simplifications. Nevertheless, the number of future scenarios to consider is still exponential in the number of possible actions (branching factor) and the length of the time horizon (search depth). A brute-force tree search over all possible plans is only feasible when the action space is discrete and small, the time horizon is short, and the time discretization is coarse.

[0006] The Multi-Policy Decision-Making (MPDM) framework is useful for this kind of planning problem because the computation time is linear in both the number of policies and the length of the planning horizon. Instead of planning in an action space and directly considering each possible control input, MPDM only considers planning in a policy space and choosing from high-level closed-loop policies that encode domain-specific behavior. MPDM handles the uncertainty of other dynamic agents by sampling their states and assuming that they are also following the policy, again bounding the computational complexity. By having policies that encode the range of behaviors that make sense for both the self-agent (the agent we are planning) and the other agents, MPDM exploits prior domain knowledge to avoid exploring highly unlikely and unrealistic parts of the full search tree. Policies can also be used for both discrete and continuous action spaces. Besides being limited to both the self-agent and the other agents following the policy, MPDM is also limited in that it does not consider the possibility of switching policies within the planning horizon. This makes certain large-scale behaviors, such as an autonomous vehicle overtaking another vehicle and then returning to the autonomous vehicle's original lane, much trickier to handle.

[0007] Efficient Uncertainty-Aware Decision-Making (EUDM) extends MPDM to help overcome this limitation by using tree search to allow at most one policy change at some future time in the planning horizon, and also using heuristics to identify situations where there are obstacle agents that may lead to dangerous situations. This helps EUDM to more effectively rule out uncertainties in the initial states and plans of other agents. Even with policies, the number of possible initial belief states is still exponential in the number of obstacle vehicles to plan for.

[0008] The present disclosure further improves MPDM to overcome the limitations of single policy modifications and the need to use heuristics in severe situations by combining insights from both MPDM and Monte Carlo Tree Search (MCTS), along with additional novel modifications that leverage a unique cost structure to focus on safety in autonomous driving and other similarly structured tasks.

[0009] This section provides background information relevant to the present disclosure that is not necessarily prior art. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] "The KL-UCB algorithm for Bounded Stochastic Bandits and Beyond" in Proceedings of the 24th Annual Conference on Learning Theory JMLR Workshop and Conference Proceedings 2011 [Non-Patent Document 2] "Efficient Uncertainty-aware Decision-making for Automated Driving using Guided Branching" 2020 IEEE International Conference on Robotics and Automation 2020 Summary of the Invention

[0011] This section provides a general overview of the disclosure, but is not a comprehensive disclosure of its complete scope or all of its features.

[0012] A computer-implemented method for issuing commands to controlled objects in a monitored environment is presented. The method includes receiving an initial state estimate for a controlled object and one or more monitored objects in the monitored environment, the initial state estimate including state elements indicative of a position of each object, a velocity of each object, and an intent of each object, and constructing a policy tree for evaluating actions to be taken by the controlled object during the evaluation period, such that each level of the policy tree represents a time interval during the evaluation period, each edge of the policy tree represents a policy to be followed by the controlled object, and each node of the policy tree stores an indication of an outcome for the controlled object following the policy defined by a path to a given node, the policy being selected from a set of possible policies. The method includes constructing an initial state estimate, generating one or more state estimates for the controlled object and one or more monitored objects from the initial state estimate, each state estimate in the one or more state estimates including a state element indicating a position of the respective object, a velocity of the respective object, and an intent of the respective object, generating one or more state estimates for the controlled object and one or more monitored objects from the initial state estimate, each state estimate in the one or more state estimates including a state element indicating a position of the respective object, a velocity of the respective object, and an intent of the respective object, evaluating results of one or more paths in a policy tree using the one or more state estimates, selecting a given path in the policy tree having a best result for the controlled object, the given path indicating a set of policies to be followed by the controlled object during the evaluation period, and issuing commands to the controlled object in accordance with the set of policies.

[0013] A policy tree may be constructed prior to evaluating the outcome of one or more paths in the policy tree, hi some embodiments, the outcome of one or more paths in the policy tree is evaluated using a Monte Carlo tree search.

[0014] In one aspect, the outcome of one or more paths in a policy tree is evaluated by evaluating a given path in the policy tree with a given state estimation that yields a cost at each node in the given path according to a cost function, and assigning an expected cost to each node in the given path, where the expected cost at a given node in the given path is determined by calculating an average expected cost at each leaf node dependent on the given node, and setting the expected cost of the given node equal to the average of the average expected costs at each leaf node dependent on the given node.

[0015] In another aspect, the outcome of one or more paths in the policy tree may be evaluated by evaluating a given path in the policy tree with a given state estimation that yields a cost at each node in the given path according to a cost function, and assigning a marginal expected cost to each node in the given path, where the marginal expected cost at a given node is set to the average of the marginal cost resulting from evaluating the state estimation at the given node plus the marginal expected cost from a particular child node of the given node, such that the marginal expected cost of the particular child node is minimum among the child nodes of the given node.

[0016] In yet another aspect, outcomes of one or more paths in the policy tree are evaluated by evaluating each child node of a root node of the policy tree with a first state estimation selected from the multiple state estimates before evaluating the paths in the policy tree using another state estimation different from the first state estimation.

[0017] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

[0018] The drawings described herein are merely for illustration of selected embodiments and not for illustration of all possible implementations, and are not intended to limit the scope of the present disclosure. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates a computer-implemented method for issuing commands to controlled objects in an autonomous driving scenario. [Figure 2A] FIG. 1 is an example policy tree diagram showing true marginal and intermediate costs. [Figure 2B] FIG. 13 is an example policy tree showing sampled intermediate costs. [Figure 2C] FIG. 1 is an example policy tree diagram illustrating a typical expected cost rule. [Figure 2D] FIG. 13 is an example policy tree diagram illustrating an alternative expectimax expected cost rule. [Figure 2E] FIG. 13 is an example policy tree diagram illustrating the lower bound expected cost rule. [Figure 2F] FIG. 13 is an example policy tree diagram illustrating the marginal expected cost rule. [Diagram 3] 13 is a graph showing a parameter sweep of the marginal action cost (MAC) and UCB constants for each expected cost rule, showing that the "classical" rule performs best. [Figure 4] Figure 13 shows a parameter sweep of UCB constants per UCB expected cost rule while using MAC for final action selection. As the UCB value increases, we can see that the performance of each rule approaches that of uniform / pure search. [Diagram 5] This graph shows the parameter sweep of a Monte Carlo trial for each UCB expected cost rule while using MAC for final action selection. MAC achieves lower regret faster than the other rules. Uniform search also performs surprisingly well, thanks to the use of the max-robust child rule to make the final decision in a timely manner. [Figure 6]Graph showing parameter sweeps of Monte Carlo trials for each UCB variation, using MAC for expected cost and final action selection. All of the improved rules outperform UCB in most cases, by roughly the same margins that UCB outperforms uniform search. KLUCB consistently performs best, and KL-UCB+ performs roughly similarly. [Figure 7] 8 is a plot of relative regret (normalized by the no-replica case), particle iteration constant, and number of trials, showing that particle iteration is strictly beneficial at least up to 256 trials, with a maximum regret reduction of about 10%. Note that all cases with 1,024+ trials have very low absolute regret (see FIG. 8). [Figure 8] Figure 1 shows an ablation study of our method, showing the benefits of starting with traditional MCTS using UCB and maximally robust children, then adding KL-UCB, marginal action cost (MAC), and finally particle iteration. Our fully enhanced method performs better in all ablation cases. [Figure 9] Figure 1 shows the MCPTDM overtaking vehicles and keeping distance from other vehicles in a simulated road environment. Vehicle 4 stops in front of vehicle 7, which stops in front of the ego vehicle (number 0). The ego vehicle moves to the left lane, overtakes vehicle 7, and keeps a small distance behind vehicle 4. It can be seen how the MCPTDM prefers to perform a tactical overtaking to move forward and also to keep a distance from vehicle 4 in case the other vehicle behaves erratically. The ego vehicle is colored green, while the obstacle vehicles are blue when moving or gray when stationary. The Monte Carlo trial is shown by its forward simulated traces, which are dark red for traces that lead to collisions, pink for traces that are somewhat unsafe, and green for traces that are safe. The frames increment by 1 second from left to right. [Figure 10]Figure 10 shows (compare with Figure 9) that MCPTDM experiences a collision in a simulated road environment. When vehicle 11 stops in front of the ego vehicle, vehicle 9 starts to make an unsafe lane change into the right lane from behind, which the ego vehicle cannot avoid. If the ego vehicle had used a replanning rate faster than 4Hz, this collision could have been avoided. From the forward simulated trace in the penultimate frame, it appears that only the scenario where vehicle 11 accelerates tries to avoid this collision first, since the intelligent driver model of the ego vehicle requires the ego vehicle to maintain a certain following distance. The frames are incremented by 0.5 seconds from left to right. [Figure 11] Figure 13 shows the performance of MCPTDM ablations without particle iterations, and furthermore, by evaluating MCPTDM ablations using a "typical" expected cost estimate rather than marginal action cost. Both improvements are significant. [Figure 12] 1 shows a final comparison of MCPTDM with EUDM (with and without the CFB heuristic) and MPDM. MCPTDM achieves significantly lower final cost or significantly lower computation time than EUDM or MPDM. [Figure 13] 13 is a graph showing a comparison of final safety costs alone (lower is better) between each method. At all computation times, MPDM is only slightly less safe than MCPTDM. As computation times increase, EUDM is also very similar. Compare with FIG. 14 and the plot of efficiency costs alone. [Figure 14] 13 shows a comparison of final efficiency costs alone (lower is better) between each method. MCPTDM quickly degrades efficiency (for better safety) and yet keeps efficiency costs relatively low as the computational budget increases. Compare with FIG. 13 and the plot of safety costs alone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. The example embodiments are provided so that the disclosure will be thorough and will fully convey the scope to those skilled in the art. Numerous specific details are described, such as examples of specific components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that the example embodiments may be embodied in many different forms, and should not be construed as limiting the scope of the present disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0021] Monte Carlo Policy Tree Decision-making (MCPTDM) is examined using synthetic scenarios and experiments that model an abstract form of an autonomous driving scenario. After describing the scenario in detail, the disclosure examines the effect of modifying the expected cost rule used by the upper confidence bound (UCB) to balance the exploration-exploitation tradeoff and to select the final best action. The disclosure also examines the effect of various improvements to the UCB. Finally, the disclosure explores the idea of ​​fairness with particle iterations, which helps to mitigate the effect of "unlucky" initial conditions, where poor outcomes can be attributed to initial conditions rather than the inherent plan being evaluated. In an autonomous driving situation, for example, an "unlucky" particle may include a nearby vehicle that intends to corral the ego vehicle while another vehicle performs a risky lane change, and once corralled, it does not matter which policy the ego vehicle chooses, even if the probability of this random cooperation is very low.

[0022] For illustrative purposes, the present disclosure considers an abstract version of an autonomous driving task with a choice of five policies (or actions), an evaluation period (i.e., time horizon) divided into four segments or time intervals, and costs associated with avoiding collisions and moving forward in a close call. It is readily understood that more or fewer time intervals, as well as more or fewer policy options, may be implemented. It is also contemplated that costs may be assigned based on other driving-related events or may otherwise vary depending on the application.

[0023] The costs associated with moving forward are likely to be relatively smooth, while the costs related to safety and potential collisions are more discontinuous. To model more complex cost distributions, the present disclosure uses a mixture of two Gaussians in one illustrative embodiment. The Gaussian mixture uses a compactly approximating real-world event prior. Other types of probability models are also contemplated by the present disclosure.

[0024] Furthermore, in autonomous driving scenarios, a lot of uncertainty lies in the initial beliefs about the behavior and intentions of other vehicles. A unique initial condition that may be dangerous for one self-agent policy is likely to be dangerous for other policies as well. To model this correlation of “dangerous situations”, the initial conditions are stored in what we call belief particles. When the same belief particle is propagated through different paths in the tree, it should be considered as correlated responses.

[0025] FIG. 1 provides an overview of a computer-implemented method for issuing commands to controlled objects, such as autonomous vehicles in an autonomous driving scenario. As a starting point, initial states of objects in a monitored environment are determined, as shown at 12. The objects include the autonomous vehicle (i.e., the controlled object) and other monitored objects in the monitored environment, such as pedestrians, other vehicles, etc. The initial state estimate includes state elements for each of the controlled and monitored objects. In an autonomous driving scenario, the state elements for each object include the position of the respective object, the velocity of the respective object, and an indication of the intent of the respective object. Other types of state elements can also be modeled.

[0026] A policy tree is used to evaluate actions taken by a controlled object during an evaluation period. Each level of the policy tree represents a time interval during the evaluation period. For example, the evaluation period may be divided into four time intervals, each time interval being one second within the four second evaluation period. Each edge of the policy tree represents a policy to be followed by the controlled object, and each node of the policy tree stores an indication of the outcome for the controlled object following the policy defined by the path to the given node.

[0027] The policy is selected from a set of possible policies. In one example, the set of possible policies includes maintaining speed in the current lane, changing to the right lane, changing to the left lane, and slowing down. In another example, the set of policies includes maintaining speed in the current lane, accelerating in the current lane, changing to the right lane while maintaining speed, changing to the right lane while accelerating, changing to the left lane while maintaining speed, and changing to the left lane while accelerating and decelerating. These policies are merely exemplary, and the disclosure is not limited to these particular policies or combinations thereof.

[0028] At 14, one or more state estimates are generated for the controlled object and one or more monitored objects from the initial state estimate. Again, each state estimate includes state elements for the controlled object and for each of the monitored objects, the state elements for each object including a position of the respective object, a velocity of the respective object, and an indication of the intent of the respective object.

[0029] In one example embodiment, each node is assigned a random mean μ i , standard deviation σ i , and the mixture weights w i A random cost probability distribution is assigned that is a mixture of two Gaussian distributions with weights that sum to 1. Because this model only includes positive costs, these Gaussian costs are bounded in the range [0,2μ i ] is saturated to

[0030] When performing a trial, a "belief particle" (i.e., a state estimate) is sampled from the initial state estimate so that all costs of using this belief particle are correlated. This can be done by sampling z-scores from a standard Gaussian distribution (z-scores are normalized Gaussian samples, where z=(x-μ) / σ). A weight threshold t between 0 and 1 that will be used to select between the mixture Gaussian components is also sampled. These z-scores, and the threshold z from the known belief particles are sampled. 1 , z 2 , t, the costs c of all node distributions should be correlated and deterministic.

number

number

[0031] Next, at 15, the outcome of one or more paths in the policy tree is recursively evaluated using one or more state estimates. For a given path and a given state estimate, a forward simulation is performed. The forward simulation produces a cost at each node in the given path according to a cost function. A simplified cost function is described below. In this example, the cost function is defined in terms of the velocity of the controlled object, the target velocity of the controlled object, and the minimum distance between the controlled object and other monitored objects. Different and more sophisticated cost functions are contemplated by the present disclosure. The calculated costs are stored in the associated node. It is readily understood that the costs may be stored in different ways. For example, each cost may be stored individually in a given node, or the costs may be stored cumulatively with a total number of costs stored in an accumulator. Additionally, the costs may be stored as marginal costs as further described below.

[0032] Continuing with reference to FIG. 1, a given path in the policy tree having the best outcome for the controlled object is selected as shown at 16, the given path indicating a set of policies to be followed by the controlled object during an evaluation period. In one example, the path having the best outcome for the controlled object is selected by identifying a child node of the root node of the policy tree having the smallest marginal expected cost as described further below, and executing the policy associated with the edge extending between the root node and the identified child node. Other path selection methods are also included within the broader aspects of the present disclosure. Finally, at 17, a command is issued to the controlled object according to the set of policies.

[0033] In some embodiments, the entire policy tree is constructed before evaluating the results of one or more paths within the policy tree, while in other embodiments, the policy tree is constructed dynamically while evaluating the results of one or more paths, for example using a Monte Carlo tree search.

[0034] To evaluate different paths in the policy tree, an expected cost is assigned to each node along the path being evaluated. Different expected cost rules may be applied to determine and assign expected costs to nodes in the policy tree. In one example, expected costs are assigned to nodes along the path being evaluated in the policy tree. More specifically, the expected cost at a given node in a given path is determined by calculating the average expected cost at each leaf node that depends on the given node, and setting the expected cost of the given node equal to the average of the average expected costs at each leaf node that depends on the given node. In another example, a marginal expected cost is assigned to nodes along the path, as further described below. Other expected cost rules may be considered within the scope of the present disclosure.

[0035] Marginal action costs (MACs) are explained further below and allow for more detailed exploration of the search tree compared to just using terminal costs (i.e., costs at leaf nodes). To put this idea in context, a common application of Monte Carlo tree search (MCTS) is for board games such as Go, with many turns, large branching factors, determinism, and rewards / costs that are assigned only when a termination condition (e.g., win / lose) is reached. By comparison, in a non-deterministic autonomous vehicle scenario, there may be long-term goals (e.g., a destination to reach), but the overriding goal of the planner is to remain safe at all times. To this end, marginal action costs allow the search to distinguish, for example, between a collision at depth 1 and a collision at depth 4, where a high collision cost can effectively prune the entire subtree below the collision. This is only possible when the collision can be attributed to a unique node.

[0036] The expected costs assigned to nodes may be used at two different points in MCTS. In the first case, they are input to the UCB selection algorithm to guide the search exploitation trade-off during the search. In the second case, they may be used for final action selection to select the final top action to execute after the computational budget for an MCTS trial has been met. Besides using expected costs, a common alternative is to choose the most visited action as the final choice. Since slight improvements may be found by continuing the search until both of these measures are met, a variation of this combined "maximally robust child" can be used to impose a 20% limit on the number of additional trials that may be performed. This explicit limit is only necessary because some of the parameter sweeps performed include degenerate cases that do not converge.

[0037] Traditionally, the expected cost assigned to a node is usually the average of each trial that passes through the node.

number

number

[0038] Rather than simply taking the average of all leaf costs from all child nodes, we can optimistically take the best (lowest) child cost for each controlled choice by simply taking the average over multiple stochastic costs at the leaf nodes. Thus, expected costs bubble up from the bottom of the tree.

number

[0039] With known intermediate costs, it can be observed that in some cases the above rule is too optimistic. For example, imagine a node corresponding to a risky action that has a 50% chance of observing a high cost and has two child nodes, neither of which imposes any cost. If we run a trial for each of these children and get a high cost for one and zero for the other, we should optimistically choose the best option and assign an expected cost of zero to the parent node, even though the true expected cost was higher. Nevertheless, when the intermediate costs per node are known, it is known that a high cost can be blamed on the parent node and not on any of the children. We cannot be fooled into believing that there is a low-cost path for one of the children, since the parent cost applies to both. A new expected cost rule can be devised that takes the cost of the best child and applies a floor on the average partial / intermediate cost of the parent node.

number

number

[0040] Taking this idea of ​​using additional information from the intermediate costs even further, we can directly calculate and assign a marginal cost to each node, and use the sum of the average marginal costs along the best paths through the tree as the total expected cost.

number

number

[0041] In an example embodiment, a Monte Carlo tree search is used to evaluate the policy tree, for example as shown in Algorithm 1 below. The input to the algorithm is an initial state s 0 , the uncertainty belief b, and the policy set P. Other variables are p for the policy, s for the belief sample / initial condition particle, n' for the child node of n, m for the marginal action cost, m for the set of marginal action costs observed by node n, n , and c for the expected cost of node n n In practice, implementations of this algorithm preferably limit the number of particles that are iterated over.

number

[0042] At the start of each Monte Carlo trial, a set of initial conditions is sampled. This particle then takes several paths through the tree, affecting the final cost of the trial. Some of these particles may be lucky or unlucky, making the actions and paths they take appear unreasonably better or worse than they actually are. For example, if all obstacle vehicles simultaneously avoid or crowd the ego vehicle, then there will either be no collision possible or a collision will be unavoidable, regardless of the ego vehicle's policy choice. The idea is that with enough trials and particles, these effects will eventually be reasonably averaged out, yielding a good result. Nevertheless, when the number of trials is computationally limited, it can be useful if particles are deliberately iterated along different paths through the tree to reduce any bias resulting from this good or bad luck.

[0043] In an example embodiment, every particle, the path every particle takes, and the terminal cost of every particle are recorded. Because the initial conditions are compact, this does not amount to a significant amount of information. In the ChoosePolicy function, the policy tree is recursively evaluated using the Recurse function until the number of trials exceeds a computation limit. When the computation limit is exceeded, the policy (or path in the policy tree) with the best result is selected. In an example embodiment, a policy is chosen by identifying the child node of the root node of the policy tree that has the smallest marginal expected cost, and executing the policy associated with the edge extending between the root node and the identified child node.

[0044] In the Recurse function, the child nodes to consider are selected using the KL-UCB algorithm. For more information on the KL-UCB algorithm, see "The KL-UCB algorithm for Bounded Stochastic Bandits and Beyond" in Proceedings of the 24th International Conference on Computational Biology and Mathematics, Vol. 16, No. 1, 2015, pp. 171-175, 2015. thThe method can be found in the Annual Conference on Learning Theory JMLR Workshop and Conference Proceedings 2011, which is incorporated by reference in its entirety. Other types of upper bound confidence algorithms and variations thereof are also contemplated by this disclosure.

[0045] After the top action is chosen for the next trial, we first check if there is a particle that has taken this path before. If so, we iterate over this particle rather than sampling a new particle. If there are multiple particles, we choose the one with the highest cost. Note that we could perform particle iteration at any depth in the tree, but we have found it to be worthwhile only at the top. The remainder of the function involves performing a forward simulation for the selected child node with the particle, storing the marginal cost, and also bubbling the marginal cost up from the child node through the policy tree. It is noted that the Recurse function terminates when the node is a leaf node in the policy tree.

[0046] Once the computational budget becomes large enough, it is no longer necessary to iterate particles for plausibility. Instead, it may be more effective to just draw new particles to better filter out the uncertainty. For a smaller number of Monte Carlo trials, it helps to iterate as many particles as possible, but for a larger number, it may be best not to iterate at all. We found that the best number of particles to iterate appears to be inversely proportional to the total number of trials in the budget. This means that it would be best to have an estimated budget before starting so that it can be performed at any time. The iteration constant is used to set the maximum number of particle iterations to perform, as the iteration constant divided by the number of trials in the budget.

[0047] Various parameter sweeps were performed to evaluate and compare different expected cost rules, UCB variations, and particle iterations. Instead of sweeping Monte Carlo trials, we excluded the number of trials with 2^10, ranging from 8 to 4,096. Enough complete runs of the algorithm were performed to show significant results in the figures, and error bars indicate the mean plus or minus one standard deviation (standard error).

[0048] For expected cost rules, we start by sweeping the UCB constant (a free parameter) for each expected cost rule. Figure 3 shows that typical and MAC rules consistently outperform the Expectimax and Lower Bound rules, and that these differences persist even for large UCB constants that may encourage nearly pure exploration. We judge these differences to reflect the effect of expected cost rules on final action selection, not on exploration.

[0049] To tease out how the expected cost rule affects UCB and final action selection, the experiment was repeated using marginal action costs for final action selection, but still varying the expected cost rule for UCB. In Figure 4, the hypothesis is confirmed that the MAC rule is most important for final action selection, and that most of the expected cost rules perform similarly in UCB. A uniform (pure search) rule is included for comparison.

[0050] As can be seen in Figure 5, the expected cost rules are also compared by their computational budget (number of Monte Carlo trials). Each rule uses the best UCB constant for the rule, as found in Figure 4, and all use the marginal action cost for final action selection. Note that the marginal action cost converges much faster than the other rules, near zero mean regret.

[0051] Recognizing that there are many enhancements and alternatives to UCB, it is desirable to see if one of these should be able to improve performance. To this end, variants of UCB were implemented, including UCB-V

[39] , UCB(δ)

[40] , KL-UCB, and the variation KL-UCB+

[41] . Parameter sweeps were implemented for each UCB variation to select the parameters that produced the lowest regret for each, and then compared. In Figure 6, it can be seen that each of the improved variations performs somewhat similarly and significantly better than UCB, but the relative differences begin to widen when the number of trials is maximized. The KL-UCB rule is used for the remainder of this disclosure due to this advantage.

[0052] In Figure 7, the iteration constant is swept and the relative regret is shown by normalizing with the regret for cases without iterations. We can see that for most of the cases, the improvement saturates when particles have been iterated as many times as possible. Intuitively, we should expect particle iterations to become less important as the number of particles increases, since a large number of particles should mitigate the effect of individual "unlucky" particles. This behavior is seen for 1,024 Monte Carlo trials in the experiment. As the relative regret is plotted, the smaller the absolute regret, the larger the error bars. For an iteration constant of 2 16 See the trend of absolute regret in Figure 8 where is used.

[0053] Finally, an ablation study in Figure 9 is conducted to compare traditional MCTS search with UCB and “maximally robust child” final action selection with an enhanced method that adds KL-UCB, MAC expected cost estimation, and finally particle iteration. Each addition significantly reduces the regret, albeit at different points along the curve.

[0054] For further evaluation, an autonomous driving scenario similar to that proposed by Zhang et al. in "Efficient Uncertainty-aware Decision-making for Automated Driving using Guided Branching" 2020 IEEE International Conference on Robotics and Automation 2020 is adopted, but with only two lanes going in a single direction (see Figures 8 and 9). The scenario uses a bicycle model, an intelligent driver model, and pure pursuit lateral control for all vehicles, with five policies: keep left lane, accelerate left lane, keep right lane, accelerate right lane, or decelerate. The vehicle performs a lane change maneuver by choosing a policy for a lane different from the current lane.

[0055] Besides the ego vehicle, 13 "obstacle" vehicles are simulated, which are removed and reinstated to maintain 13 vehicles within a certain distance of the ego vehicle. This number of vehicles ensures that there can be complex interactions between multiple other vehicles both in front of and behind the ego vehicle, and also ensures that the environment is not overcrowded. Each obstacle vehicle is parameterized with a random (within some range) preferred speed, acceleration, and following time to introduce some variability and uncertainty in the behavior of the obstacle vehicle. Every 0.2 seconds, each obstacle vehicle has a small chance (5% chance per second) to randomly choose a new policy. Since the change of obstacle vehicles occurs randomly, the policy used by the obstacle vehicles will first check whether the next lane is clear at least half the length ahead and behind the vehicle before making a lane change maneuver. The policy used by the ego vehicle does not perform this check and can therefore be more flexible.

[0056] The ego vehicle attempts to maintain a target speed safely and smoothly by minimizing a cost function that incorporates speed, safety, and control inputs. C vel=|vv target (8)

number

[0057] Each obstacle vehicle may or may not intend to perform a lane change maneuver. From the ego-agent's point of view, this is a hidden state that must be estimated in order to perform a forward rollout.

[0058] For simplicity, a stateless heuristic method for belief estimation is implemented based on thresholds on the direction the vehicle is pointing, the vehicle's position in its lane, and the vehicle's speed relative to the vehicle in front of it. There is room for improvement in this belief estimation, but this should not affect the validity of the method comparison.

[0059] A typical multi-policy decision-making (MPDM) comparison takes a sample (according to a computational budget) from the belief state, and then a closed-loop simulates the sample forward through each of the five policy choices for 8 seconds. Finally, the policy with the lowest average cost is chosen.

[0060] Efficient Uncertainty-Aware Decision-Making (EUDM) is an extension of MPDM that includes both an eigentree search through policies and a heuristic for selecting the belief samples that represent the most critical / risky cases.

[0061] The tree search used by EUDM, the "Domain-Specific Closed-Loop Policy Tree", allows for only one policy change in the planning horizon, and this change must occur below the root node of the tree. A policy tree with depth 4 and each layer taking 2 seconds is used, so that the total horizon is 8 seconds. Since the policy change must occur after the root node, EUDM has a built-in history and will only actually change the policy 2 seconds after making this decision the first time if EUDM still wants to. The current EUDM selected policy can be used as input to a separate "Space-Time Semantic Corridor" trajectory generation module, which produces the actual behavior for the ego vehicle. This preliminary module allows the ego vehicle to recognize risks and react to changing conditions, even with 2 seconds of policy history.

[0062] The EUDM has been modified to allow for policy switching at any time, including immediately, which is a departure from the original EUDM methodology but improves the performance of the EUDM.

[0063] The heuristic used by EUDM, called conditional focused branching (CFB), selects nearby obstacle vehicles, filters only those whose policies are uncertain, then performs an open-loop forward simulation for each belief policy, and finally produces a set of most likely belief samples formed by the Cartesian product of the vehicle and the policy for each obstacle vehicle deemed to be dangerous by the open-loop simulation. All non-hazardous vehicles are assigned their most likely policy.

[0064] In this implementation of EUDM, an open-loop forward simulation is performed by giving only self and obstacle vehicles under the inspection dynamic policy, simulating all other vehicles using only constant speed. The obstacle vehicles are ordered according to their "risk", which is the difference between the minimum and maximum cost from each of the policy choices, and then we choose the four most dangerous vehicles. We form a Cartesian product of these dangerous obstacle vehicles with their policies, and then we finally select the most probable scenarios according to our beliefs and weight them according to their probabilities. As many scenarios as allowed are taken within the computational budget.

[0065] The final method for application to automated driving scenarios uses improvements from the synthetic experiments described above: Marginal Action Cost (MAC) expected cost estimation and particle iteration. KL-UCB and "maximally robust child" selection are used as in the previous experiments. Furthermore, when expanding a node in the search tree, we first explore children with the same policy as the parent, since in most cases the ego vehicle will maintain its current policy.

[0066] For the experiments, we perform 16,384 runs per method to obtain significant results. To complete all these runs within a reasonable time frame, we limit each run to a simulated 30 seconds and reschedule at 4Hz. Runs are performed on a single-threaded 2.5GHz Intel Xeon E5-2640 per run, allowing multiple runs to be performed in parallel.

[0067] To make a reasonable comparison, we plot the final cost observed in each run (without discounting) against the 95% computational latency - that is, 95% of the replanning periods for a 30 second run will have a latency less than this. The closer a method is to the bottom left corner (closer to zero cost and zero time), the better it is.

[0068] Over a total of 589,824 30-second long runs to perform all of these figures, there were a total of 2,573 collisions (0.44%). The error bars are larger in some of the figures because collisions are relatively rare and also lead to much higher final costs.

[0069] First, an ablation is performed to ascertain the separate contributions of MAC and particle iterations. A parameter sweep was performed to determine reasonable constants for KLUCB (for the full method) and for particle iterations, and it was found to perform best with as many iterations as possible. In Figure 11, the results for both MAC and particle iterations show significant improvement.

[0070] Finally, a comparison is made between the proposed MCPTDM method and the baseline methods of MPDM and EUDM in Fig. 12. MCPTDM achieves significantly lower cost during similar computation time. The cost function is composed of safety cost (to avoid collision and overshooting), efficiency cost (to approach the target speed), and maneuvering and acceleration cost (to minimize the control input), with safety and efficiency being the most important. Note that although only safety and efficiency costs are compared in Fig. 13 and Fig. 14, most of the lower cost improvement of MCPTDM over MPDM and EUDM comes from keeping efficiency without sacrificing safety. Intuitively, it makes sense that in most cases, improved safety (lower safety cost) comes along with worse efficiency (lower average speed and higher efficiency cost).

[0071] The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0072] Some sections above present the techniques described herein in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, will be understood to be implemented by computer programs. Further, it has proven convenient at times to refer to these arrangements of operations as modules or by function names, without loss of generality.

[0073] Unless specifically stated otherwise, and as is apparent from the above discussion, throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like are understood to refer to the actions and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical (electronic) quantities in the computer system memory or registers or other such information storage, transmission or display device.

[0074] Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm, it being noted that the described process steps and instructions may be embodied in software, firmware, or hardware, and when embodied in software, may be downloaded to reside on and operated from different platforms for use by a real-time network operating system.

[0075] The present disclosure also relates to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes, or may comprise a computer selectively activated or reconfigured by a computer program stored in a computer-readable medium accessible by the computer. Such a computer program may be stored on a tangible computer-readable storage medium, such as, but not limited to, a floppy disk, an optical disk, a CD-ROM, any type of disk including a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions and coupled to a computer system bus, respectively. Furthermore, the computers referred to herein may include a single processor or may be architectures employing a multi-processor design for increased computing power.

[0076] The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The structure required for a variety of these systems, along with equivalent variations, will be apparent to those skilled in the art. Further, the present disclosure is not described with reference to any particular programming language. It is understood that a variety of programming languages ​​may be used to implement the teachings of the present disclosure as described herein.

[0077] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure. Individual elements or features of a particular embodiment are not generally limited to this particular embodiment, but may be interchangeable, where applicable, and may be used in selected embodiments even if not specifically shown or described. The same may likewise be modified in many ways. Such modifications should not be considered as departures from the present disclosure, and it is intended that all such modifications be included within the scope of the present disclosure.

Claims

1. 1. A computer-implemented method for issuing commands to controlled objects in a monitored environment, comprising: receiving an initial state estimate for the controlled object and one or more monitored objects in the monitored environment, the initial state estimate including state elements indicative of a position of the respective object, a velocity of the respective object, and an intent of the respective object; constructing a policy tree for evaluating actions to be taken by the controlled object during the evaluation period, such that each level of the policy tree represents a time interval during the evaluation period, each edge of the policy tree represents a policy to be followed by the controlled object, and each node of the policy tree stores an indication of the outcome for the controlled object of following a policy defined by a path to a given node, the policy being selected from a set of possible policies; generating one or more state estimates for the controlled objects and the one or more monitored objects from the initial state estimates, each state estimate in the one or more state estimates including a state element indicative of a position of the respective object, a velocity of the respective object, and an intent of the respective object; using the one or more state estimates to evaluate outcomes of one or more paths in the policy tree; selecting a given path in the policy tree having a best outcome for the controlled object, the given path indicating a sequence of policies to be followed by the controlled object during the evaluation period; issuing commands to said controlled objects in accordance with said set of policies; A method comprising:

2. The method of claim 1 , wherein the policy tree is constructed prior to evaluating the results of one or more paths within the policy tree.

3. The method of claim 1 , further comprising: evaluating outcomes of the one or more paths in the policy tree using a Monte Carlo tree search.

4. The method of claim 1 , wherein the policies in the set of possible policies include maintaining speed in a current lane, changing to a right lane, changing to a left lane, and slowing down.

5. Evaluating the outcome of one or more paths in the policy tree comprises: evaluating a given path in the policy tree using a given state estimate that yields a cost at each node in the given path according to a cost function; assigning an expected cost to each node in the given path, wherein the expected cost at a given node in the given path is determined by calculating an average expected cost at each leaf node dependent from the given node, and setting the expected cost of the given node equal to the average of the average expected costs at each leaf node dependent from the given node; The method of claim 1 further comprising:

6. 6. The method of claim 5, further comprising storing one of the costs associated with nodes at the corresponding node on the given path or a marginal cost associated with nodes at the corresponding node on the given path.

7. Evaluating the outcome of one or more paths in the policy tree comprises: evaluating a given path in the policy tree using a given state estimate that yields a cost at each node in the given path according to a cost function; assigning a marginal expected cost to each node in the given path, the marginal expected cost at the given node being set to the average of the marginal cost resulting from evaluating a state estimate at the given node plus the marginal expected cost from the particular child node of the given node, such that the marginal expected cost of the particular child node is the smallest among the child nodes of the given node; The method of claim 1 further comprising:

8. 8. The method of claim 7, further comprising the step of: identifying a child node of a root node of the policy tree having a minimum marginal expected cost; and selecting a given path in the policy tree having a best outcome for the controlled object by executing a policy associated with the edge extending between the root node and the identified child node.

9. 2. The method of claim 1, wherein evaluating outcomes of one or more paths in the policy tree comprises evaluating each child node of a root node of the policy tree with the first state estimation selected from the plurality of state estimations before evaluating paths in the policy tree using another state estimation different from a first state estimation.

10. 1. A computer-implemented method for issuing commands to controlled objects in a monitored environment, comprising: receiving, by a computer processor, an initial state estimate for the controlled object and one or more monitored objects in the monitored environment, the initial state estimate including state elements indicative of a position of each of the objects, a velocity of each of the objects, and an intent of each of the objects; constructing, by the computer processor, a policy tree for evaluating actions to be taken by the controlled object during the evaluation period, such that each level of the policy tree represents a time interval during the evaluation period, each edge of the policy tree represents a policy to be followed by the controlled object, and each node of the policy tree stores an indication of the outcome for the controlled object of following a policy defined by a path to a given node, the policy being selected from a set of possible policies; generating, by the computer processor, one or more state estimates for the controlled objects and the one or more monitored objects from the initial state estimates, each state estimate in the one or more state estimates including a state element indicative of a position of the respective object, a velocity of the respective object, and an intent of the respective object; evaluating, by the computer processor, outcomes of one or more paths in the policy tree using the one or more state estimates and a Monte Carlo tree search; identifying a child node of the root node of said policy tree having a minimum expected cost; executing a policy associated with the edge extending between a root node and the identified child node; issuing commands to the controlled objects in accordance with the executed policies; 4. A computer-implemented method comprising:

11. A non-transitory computer-readable medium, which upon execution of computer-executable instructions by a processor of a computer, receiving an initial state estimate for a controlled object and one or more monitored objects in a monitored environment, the initial state estimate including state elements indicative of a position of each of the objects, a velocity of each of the objects, and an intent of each of the objects; constructing a policy tree for evaluating actions to be taken by the controlled object during the evaluation period, such that each level of the policy tree represents a time interval during the evaluation period, each edge of the policy tree represents a policy to be followed by the controlled object, and each node of the policy tree stores an indication of the outcome for the controlled object of following a policy defined by a path to a given node, the policy being selected from a set of possible policies; generating one or more state estimates for the controlled objects and the one or more monitored objects from the initial state estimates, each state estimate in the one or more state estimates including a state element indicative of a position of the respective object, a velocity of the respective object, and an intent of the respective object; evaluating outcomes of one or more paths in the policy tree using the one or more state estimates; selecting a given path in the policy tree having a best outcome for the controlled object, the given path indicating a set of policies to be followed by the controlled object during the evaluation period; issuing commands to said controlled objects in accordance with said set of policies; A non-transitory computer-readable medium having computer-executable instructions for causing the computer to perform the method.

12. The non-transitory computer-readable medium of claim 11 , wherein the policy tree is constructed prior to evaluating results of one or more paths within the policy tree.

13. 12. The non-transitory computer readable medium of claim 11, the computer program instructions further performing: evaluating results of the one or more paths in the policy tree using a Monte Carlo tree search.

14. 12. The non-transitory computer-readable medium of claim 11, wherein the policies in the set of possible policies include maintaining speed in a current lane, changing to a right lane, changing to a left lane, and slowing down.

15. The computer program instructions further comprising: evaluating a given path in the policy tree using a given state estimate that yields a cost at each node in the path according to a cost function; assigning an expected cost to each node in the given path, wherein the expected cost at a given node in the given path is determined by calculating an average expected cost at each leaf node dependent from the given node, and setting the expected cost of the given node equal to the average of the average expected costs at each leaf node dependent from the given node; 12. The non-transitory computer readable medium of claim 11, further comprising:

16. 16. The non-transitory computer readable medium of claim 15, wherein the computer program instructions are further configured to store one of the costs associated with nodes at the corresponding node on the given path or a marginal cost associated with nodes at the corresponding node on the given path.

17. The computer program instructions further comprising: evaluating a given path in the policy tree using a given state estimate that yields a cost at each node in the path according to a cost function; assigning a marginal expected cost to each node in the given path, the marginal expected cost at the given node being set to an average of the marginal cost resulting from evaluating a state estimate at the given node plus the marginal expected cost from the particular child node of the given node, such that the marginal expected cost of the particular child node is the smallest among the child nodes of the given node; 12. The non-transitory computer readable medium of claim 11, further comprising:

18. 20. The non-transitory computer-readable medium of claim 17, wherein the computer program instructions further perform: identifying a child node of a root node of the policy tree having a minimum marginal expected cost; and selecting a given path in the policy tree having a best outcome for the controlled object by executing a policy associated with the edge extending between the root node and the identified child node.

19. 12. The non-transitory computer-readable medium of claim 11, wherein the computer program instructions are further configured to evaluate each child node of a root node of the policy tree with the first state estimation selected from the plurality of state estimations before evaluating a path in the policy tree using another state estimation different from the first state estimation.

Images