Vehicle trajectory control using tree search.

Abstract

Trajectory generation for controlling the movement or other behavior of an autonomous vehicle may include alternating between determining candidate actions and predicting a future state based on the candidate actions. The technique may include determining a cost associated with the candidate actions, which may include an estimate of the transition cost from a current or previous state of the vehicle to a next state. The cost estimate may be a floor or ceiling cost, and the tree search may apply the floor or ceiling cost alternately, exclusively, or according to a ratio or a changing ratio. The prediction of the future state may be based at least in part on the classification of the dynamic object of the machine learning model as being a reactive object or being an inert object, which may change how the dynamic object is modeled for the prediction.

Description

[Technical field]

[0001] The present disclosure relates to vehicle trajectory control using tree search. [Background technology]

[0002] An autonomous vehicle may not be able to navigate accurately and / or efficiently when baseline operating conditions change, such as when road markings are obscured (e.g., by snow, debris, sand), degraded (e.g., out lights, worn lane markings), and / or disabled (e.g., an obstacle partially blocks a lane, traffic signs and / or traffic cones indicate an alternate lane that is incompatible with the original lane markings). Moreover, various environmental factors and human and animal behavior may be erratic or unpredictable, which may make autonomous vehicle navigation even more challenging. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application Serial No. 16 / 889,747 [Patent Document 2] U.S. Patent Application Serial No. 16 / 581,632 [Patent Document 3] U.S. Patent Application Serial No. 16 / 682,971 [Patent Document 4] U.S. Patent Application Serial No. 16 / 703,625 [Patent Document 5] U.S. Patent Application Serial No. 15 / 963,833 [Patent Document 6] U.S. Patent Application Serial No. 16 / 407,139 [Patent Document 7] U.S. Patent Application Serial No. 17 / 184,559 [Patent Document 8] U.S. Patent Application Serial No. 17 / 006,679 [Patent Document 9] U.S. Patent Application Serial No. 16 / 872,284 [Brief description of the drawings]

[0004] The detailed description is now described with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items.

[0005] [Figure 1] FIG. 1 illustrates an autonomous vehicle and an example scenario in which lane references (whether previously mapped or detected) may not be reliable in determining commands for controlling the vehicle's movement. [Diagram 2] FIG. 1 is a block diagram of an example autonomous vehicle architecture including a guidance system for unstructured path planning. [Figure 3A] FIG. 1 is an illustrative flow diagram of an example process for generating a path for controlling an autonomous vehicle using an agent filter and / or a tree search with alternating estimated costs that may be generated from sensor data, and may include an occupancy map showing static objects and / or a map of probabilities of predicted positions and / or velocities of dynamic objects. [Figure 3B] FIG. 1 is an illustrative flow diagram of an example process for generating a path for controlling an autonomous vehicle using an agent filter and / or a tree search with alternating estimated costs that may be generated from sensor data, and may include an occupancy map showing static objects and / or a map of probabilities of predicted positions and / or velocities of dynamic objects. [Figure 3C]FIG. 1 is an illustrative flow diagram of an example process for generating a path for controlling an autonomous vehicle using an agent filter and / or a tree search with alternating estimated costs that may be generated from sensor data, and may include an occupancy map showing static objects and / or a map of probabilities of predicted positions and / or velocities of dynamic objects. [Figure 4] FIG. 1 illustrates a three-dimensional representation of a set of different candidate trajectories generated for different time steps / positions in the environment associated with a branch of the tree search discussed herein. [Figure 5A] FIG. 13 is a top-down diagram of various candidate actions that may be generated by a tree search for various action nodes at the same level of the tree search. [Figure 5B] FIG. 13 is a top-down diagram of various candidate actions that may be generated by a tree search for various action nodes at the same level of the tree search. [Figure 6A] FIG. 5B is a top-down view of two different predicted environmental states associated with a single predictive node or two different predictive nodes that may be determined based at least in part on the candidate actions shown in FIG. 5A. [Figure 6B] FIG. 5B is a top-down view of two different predicted environmental states associated with a single predictive node or two different predictive nodes that may be determined based at least in part on the candidate actions shown in FIG. 5A. [Figure 7A] FIG. 7A is a top-down view of two different candidate actions generated based on the state associated with the first predictive node shown in FIG. 6A. FIG. 7A also shows the results of agent filtering to change the classification of dynamic objects from reactive objects to inactive objects for use by the prediction and / or simulation component in generating the next level predictive nodes. [Figure 7B]FIG. 7B is a top-down view of two different candidate actions generated based on the state associated with the first predictive node shown in FIG. 6A. FIG. 7B also shows the results of agent filtering to change the classification of dynamic objects from reactive objects to inactive objects for use by the prediction and / or simulation component in generating the next level predictive nodes. [Figure 8A] FIG. 1 is a top-down view of curved roads, lane criteria, and offset criteria used to probabilistically generate actions for a tree search. [Figure 8B] 11 is a graph illustrating a sampling technique for determining an action for a tree search that may be used to ensure that a sufficient parameter space is sampled to find a suitable path. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As discussed above, due to the occurrence of abnormal behavior and changes in road conditions, it may be difficult to prepare an autonomous vehicle for all eventualities. These situations may result in the autonomous vehicle stuttering or hesitating, stopping completely when a human driver would be able to navigate the situation, and / or the need to transmit a request for help from a remote operator (or "teleoperator"). The present application relates to techniques for increasing the number of scenarios in which an autonomous vehicle can safely and effectively navigate, for example, without stopping, stuttering, or needing to request help from a teleoperator, and / or by reducing the likelihood of a collision occurring, particularly with respect to abnormal situations, and also with respect to baseline driving conditions. For example, the techniques discussed herein may reduce the occurrence of an autonomous vehicle stopping or stumbling with respect to benchmark situations such as a traffic cone pushed into the center of a lane, an object such as a vehicle blocking part of two lanes, debris on the street, complex intersections with many vehicles and pedestrians, driving through gravel areas without lane markings, etc.

[0007] The techniques discussed herein may include an autonomous vehicle guidance system that generates a path for controlling an autonomous vehicle based at least in part on a tree search technique that alternately determines candidate actions and predicts future states of an environment associated with the autonomous vehicle in response to the candidate actions. In some examples, the tree search technique may include detecting static and dynamic objects in an environment associated with the autonomous vehicle and classifying the dynamic objects as reactive or inert objects via a machine learning (ML) model. A reactive object may be more likely to change its movement in response to an action of the autonomous vehicle (as measured by a confidence output by the ML model), while the movement of an inert object may be less likely to change based on the movement of the autonomous vehicle.

[0008] The ML model may receive a track or at least a pose (e.g., position and / or orientation) associated with an autonomous vehicle, a track associated with a dynamic object, and / or sensor data associated with the dynamic object, and may be trained to output a confidence score indicating the likelihood that an action of the dynamic object will be responsive / affected by an action of the autonomous vehicle. The track may identify the detected object's past, current, and / or predicted posture, speed, acceleration, object classification (which may or may not be immutable, e.g., a classification of "vehicle" is likely to be immutable while "cyclist" may change because a cyclist may get off and become a pedestrian), etc. In some examples, if the confidence score determined by the ML model meets or exceeds a confidence threshold, the detected object may be classified as a reactive object, otherwise the detected object may be classified as an inert object. In some examples, the classification of the detected object as a dynamic object may be handled by a different ML model, such as an ML model that determines a track associated with the detected object. Nevertheless, more simply, a dynamic object may be an object that is detected as moving, having moved, and / or associated with an object classification associated with a movable object. In some examples, another ML model may determine a static object map (e.g., a map indicating space occupied by static objects and / or "free space" in the environment) and / or one or more dynamic object maps (e.g., a map indicating the likelihood that a location in the environment will be occupied by a moving object or a stationary object that may move at a future point in time). These maps may be used for the cost determination portion of the tree search performed by the vehicle guidance system.

[0009] The tree search discussed herein may alternately determine a candidate action and a predicted state of the environment associated with (e.g., at least partially responsive to) that candidate action at a future time step, another candidate action based on that predicted state of the environment, a second predicted state of the environment associated with that additional candidate action at a further future time step, and so on, for a time horizon or up to a specified number of actions. The candidate actions may indicate, for example, a trajectory for controlling the movement of the vehicle, activating emitters (e.g., turn signals, headlights, speakers) of the vehicle, and so on. Each of the candidate actions may be associated with a separate action node, and each of the predicted environmental states may be associated with a prediction node of the tree. As an initial operation, the tree search may determine a current state of the environment associated with the autonomous vehicle, which may include dynamic and / or static objects, based at least in part on the sensor data. This initial state may be associated with a root node. The root node may be a prediction node in at least one example. The root node and / or any of the other prediction nodes may identify dynamic objects as reactive objects or inert objects as part of the state of the environment indicated by the prediction node. The state of the environment may, in some examples, be represented by a data structure associated with the root node / predictor node.

[0010] Using this initial state, the tree search may determine one or more candidate actions for exploration. The candidate actions may include coarse maneuvers such as "stay in the same lane," "change lane to the left," "perform a right turn," "stop," etc., and / or fine instructions such as curves that define and / or are associated with the position, steering angle, steering rate, speed, and / or acceleration that the vehicle controller tracks. In some examples, determining one or more candidate actions for exploration may include transmitting the initial environmental state (or the state indicated by a particular prediction node of the branch being explored at a prediction node deeper than the initial node) to a planning component of the vehicle and receiving a set of candidate actions from the planning component. The planning component may be a nominal planning component of the vehicle that generates trajectories for controlling the movement and / or operation of the vehicle, as opposed to a contingent planning component that controls the vehicle during abnormal or emergency situations, although it is contemplated that the contingent planning component may additionally or alternatively provide and generate candidate actions for use by the tree search. A tree search component can associate one or more candidate actions of the set received from the planning component with an action node.

[0011] In some examples, each of the candidate actions may be associated with its own action node. For example, the candidate action may be associated with an action node subordinate to a root node that indicates the state on which the candidate action is based. In one example, each of the candidate actions may be associated with an individual action node, while in additional or alternative examples, a machine learning (ML) model may cluster the candidate actions and determine a representative candidate action to represent the two or more candidate actions of the cluster. In such examples, nodes associated with separate candidate actions of the same cluster may be merged into a single action node. The action node may identify a representative candidate action in some examples, or in other examples, the action node may indicate a representative and two or more candidate actions associated therewith. In examples where action nodes are merged, the lowest cost candidate action from among the two or more candidate actions represented by the action node may be selected for inclusion in subsequent searches and / or paths.

[0012] The tree search may transmit the candidate action and the state on which the candidate action was based (i.e., the initial state in this example) to a simulation and / or prediction system of the vehicle's perception component. The simulation system may be part of a prediction component for generating an estimated future trajectory for the object. The simulation system may determine a predicted state of the environment based at least in part on the candidate action. This may include predicting how inert and / or reactive dynamic objects will behave based at least in part on the candidate action. The predictions generated by the simulation system may be associated with future time points and may be used to generate prediction nodes that are subordinate to the action nodes associated with the candidate action and indicate predicted states of the environment that may result from implementing the candidate action.

[0013] The tree search may additionally or alternatively determine a cost associated with the candidate action. This cost may be based at least in part on a prediction of how close the candidate action will cause the autonomous vehicle to pass static and / or dynamic objects, the proximity of the autonomous vehicle to an undriveable surface, the speed, acceleration, and / or jerk associated with the candidate action, short-term goal costs (e.g., displacement / progress along a route, parking, achieving soft goals indicated by a remote operation device, i.e., goals that are not required to be met by the autonomous vehicle but are included in the cost determination to increase the likelihood that the autonomous vehicle will operate in a particular manner), etc. In at least one example, a cost may be determined for each action node, and a total cost may be determined in association with the action node, including the sum of the nodes that make up the branch of the data structure that contains the action node (e.g., the cost of the action node and the costs of all parent nodes from which the action node depends, going back to the root node).

[0014] In at least one example, determining the cost may be further based on a cost-to-go, which may be a prediction of what it would cost to move the autonomous vehicle in a particular manner (e.g., steering speed, steering angle, speed, acceleration) and / or to a particular location. For example, the cost-to-go may be an estimate that does not require running a simulation to determine a more accurate prediction of the cost-to-go, thereby increasing the number of candidate actions that may be explored and / or reducing computational resources consumed by the tree search. In some examples, the techniques discussed herein may include alternating, and / or alternating according to a ratio, between using a lower estimate of the cost-to-go and an upper estimate of the cost-to-go. For example, a first action node in a first layer may use the lower estimate, and a second action node in a second layer deeper than the first layer may use the upper estimate. The ratio may set how many layers use the lower estimate to how many layers use the upper estimate, e.g., two layers use the lower estimate before switching to using an upper estimate for two layers, two layers use the lower estimate before switching to using an upper estimate for one layer, etc. In at least one example, the lower estimate may be zero and the upper estimate may be a cost associated with a default action, such as holding the vehicle's steering angle and / or speed, driving straight at a specified speed, etc. In some examples, the default action may be determined at least in part based on the vehicle's track, perception data, etc.

[0015] The technique may additionally or alternatively include changing the status of the dynamic object from a reactive object to an inactive object or vice versa based at least in part on the latest predicted state of the environment. For example, the predicted track of the object associated with the predicted state and / or the candidate actions of the next layer may be provided as input to the machine learning model of the agent filter discussed herein, which may classify the dynamic object as being a reactive object or an inactive object, which may differ from how the object was previously classified. In other words, the predicted action of the dynamic object may cause the dynamic object to be far enough away from the autonomous vehicle and / or further candidate actions associated with the autonomous vehicle, such that the dynamic object is unlikely to change its behavior based on the action of the autonomous vehicle. Conversely, a dynamic object that was previously too far away, pointed in a particular direction, experienced environmental conditions that prevented interaction with the autonomous vehicle (e.g., a red light rather than a turn at the stop of the dynamic object), etc., that may have prevented the dynamic object from being affected by an action by the autonomous vehicle, may be more likely to be affected now or in the future. In a first example, determining the classification of the dynamic object may be performed in conjunction with determining the root node of the data structure, and this classification may persist throughout each of the layers of the data structure, however, in another example, the classification may be updated at each of the layers of the predictive node and / or every n layers, where n is a positive integer.

[0016] The techniques discussed herein may reduce the computational burden of determining a path for controlling an autonomous vehicle, such as by agent filtering (e.g., showing dynamic objects as inert or reactive objects) by alternating cost estimation instead of using simulation to determine a more accurate cost, and / or by using default control instructions to estimate an upper estimated cost instead of more intensive techniques for cost determination, such as simulation. The techniques may increase the number of candidate actions that may be explored and thus the number of paths that may be explored in making a decision on how to control an autonomous vehicle, where a path may be a sequence of candidate actions that form a continuous series. Increasing the number of candidate actions and / or paths explored may increase the granularity at which an autonomous vehicle may be controlled and may increase the number of scenarios in which an autonomous vehicle may navigate successfully and safely.

[0017] Example scenario FIG. 1 illustrates an example scenario 100 including a vehicle 102. In some examples, the example scenario 100 may be a real-world scenario and / or the example scenario 100 may be a representation of a real-world scenario modeled as a simulated scenario. In examples where the example scenario 100 is a simulated scenario, the example scenario 100 may be determined based at least in part on input received at a user interface of a computing device (e.g., a user of the computing device may define an environment, objects therein, and / or characteristics thereof) and / or the example scenario 100 may be based at least in part on log data received from one or more autonomous vehicles. The log data may be based at least in part on sensor data received at the autonomous vehicle, sensory data generated by a perception component, and / or instructions generated by a planning component. In some examples, the autonomous vehicle may store the log data and / or periodically transmit the log data to a remote computing device.

[0018] In some cases, the vehicle 102 may be an autonomous vehicle configured to operate according to the Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the entire trip without the driver (or passenger) being expected to control the vehicle at any time. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. It is contemplated that the techniques discussed herein may apply beyond robotic control, such as for autonomous vehicles. For example, the techniques discussed herein may be applied to mining, manufacturing, augmented reality, and the like. Moreover, while the vehicle 102 is shown as a land vehicle, the vehicle 102 may be a spacecraft, a watercraft, and the like. In some examples, the vehicle 102 may be represented in a simulation as a simulated vehicle. For simplicity, the discussion herein does not distinguish between simulated and real-world vehicles. As such, references to a "vehicle" may refer to a simulated and / or real-world vehicle.

[0019] In accordance with the techniques discussed herein, and examples in which the scenario 100 is a real-world example, the vehicle 102 may receive sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include position sensors (e.g., Global Positioning System (GPS) sensors), inertial sensors (e.g., accelerometer sensors, gyroscope sensors, etc.), magnetic field sensors (e.g., compasses), position / velocity / acceleration sensors (e.g., speedometer, drive system sensors), depth position sensors (e.g., LIDAR sensors, RADAR sensors, sonar sensors, time-of-flight (ToF) cameras, depth cameras, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (e.g., barometers, hygrometers, etc.). In some examples, the simulated sensors may correspond to at least one of the sensors 104 on the vehicle 102, and in the simulation, one or more of the sensors 104 may be simulated. In some examples, the simulated sensor position may correspond to the relative position of one of the sensors 104 with respect to the vehicle 102 .

[0020] The sensors 104 may generate sensor data, which may be received by a computing device 106 associated with the vehicle 102. In other examples, however, some or all of the sensors 104 and / or computing devices 106 may be separate from and / or located remotely from the vehicle 102, and data capture, processing, command, and / or control may be communicated to / from the vehicle 102 by one or more remote computing devices via wired and / or wireless networks.

[0021] The computing device 106 may include a memory 108 that stores the perception component 110, the planning component 112, the guidance system 114, and / or the controller 116. In some examples, the planning component 112 may include the guidance system 114. The perception component 110, the planning component 112, the guidance system 114, and / or the controller 116 may include one or more machine learning (ML) models and / or other computer-executable instructions. In general, the perception component 110 may determine what is in the environment surrounding the vehicle 102, and the planning component 112 may determine how to operate the vehicle 102 according to information received from the perception component 110. For example, the planning component 112 may determine the trajectory 118 based at least in part on the sensory data and / or other information, such as one or more maps, localization information (e.g., where the vehicle 102 is in the environment relative to the maps and / or features detected by the perception component 110), and / or a path generated by the guidance system 114. The trajectory 118 may be one of the candidate actions determined by the guidance system 114.

[0022] The trajectory 118 may include instructions for the controller 116 of the autonomous vehicle 102 to operate the drive components of the vehicle 102 to achieve a steering angle and / or steering rate that may result in a vehicle position, vehicle speed, and / or vehicle acceleration that follows a path generated by the guidance system. For example, the trajectory 118 may include a target direction, a target steering angle, a target steering rate, a target position, a target speed, and / or a target acceleration for the controller to follow as part of the path. For example, a coarse path generated by the guidance system 114 in accordance with the techniques discussed herein may indicate the vehicle's position, direction, speed, and / or entry / exit curvature at 500 millisecond time intervals, and a smooth path output by the guidance system 114 may include such points at 10 or 100 millisecond intervals, which may correspond to the time intervals associated with the trajectory 118. In some examples, the controller may include software and / or hardware for operating the drive components of the vehicle 102 sufficient to follow the trajectory 118 (and / or a path that may include multiple trajectories in one example).

[0023] In the example scenario 100, the autonomous vehicle 102 has received and / or determined a route 120 that defines a start location 122, an end location 124, and a curve between the start location 122 and the end location 124 (note that the curve includes straight lines and / or one or more curves). For example, the planning component 112 may have determined the route 120 based at least in part on sensor data and an end location received as part of a mission (e.g., from a passenger, from a command center). As used herein, references to "position" may include both a position and / or an attitude (e.g., a position and / or a heading / orientation of the vehicle). In some examples, the route may not include the end location 124 and may additionally or alternatively include a target position such as a target lane, a target relative position (e.g., 10 feet from the edge of the road), a target object (e.g., follow the vehicle, follow the passenger, move toward an individual calling the vehicle), etc.

[0024] As the vehicle operates to reach end location 124, autonomous vehicle 102 may encounter scenarios such as example scenario 100 where a planner that relies on lane criteria (e.g., a relative spatial designation determined at least in part based on a map and / or locating autonomous vehicle 102) to generate a path may not generate a path accurately and / or efficiently. For example, various objects (e.g., obstructing vehicle 126, toolbox 128, and fallen traffic cone 130) cumulatively block all three lanes of the road shown, which may cause another planner to stop the vehicle and / or invoke remote operations because no single lane has enough room for the autonomous vehicle.

[0025] However, the guidance system 114 discussed herein may generate the path 132 based at least in part on the environmental data 134 generated from the sensor data captured by the sensor 104. For example, the perception component 110 may generate all or a portion of the environmental data 134, which may include static data and / or dynamic data. For example, the static data may indicate the likelihood that an object is present at a location in the environment, and the dynamic data may indicate the likelihood that an object occupies or will occupy a location in the environment. In some cases, the dynamic data may include multiple frames associated with separate time steps spaced apart up to a prediction range (i.e., the maximum time / distance for which the dynamic data is predicted). In some examples, the guidance system 114 may be constantly running, i.e., the guidance system may be a nominal planning component, or in alternative examples, the guidance system 114 may be a contingency planning component, or a planning component for special situations (e.g., when the nominal planning component is not capable of finding a valid path).

[0026] The techniques discussed herein may additionally or alternatively include determining dynamic data based at least in part on a mode of operation of the vehicle. For example, the dynamic data may include a first prediction associated with a first vehicle mode and a first time, and a second prediction associated with a second vehicle mode and a first time. The vehicle modes may include mission level modes, such as picking up passengers, transporting passengers, dropping off passengers, etc., path or trajectory level modes, such as holding trajectory, slowing to a stop, shifting lanes, performing a right turn, etc., and / or signal modes, such as activating speakers, activating turn signals, turning on headlights or high beams, etc. The autonomous vehicle's actions and signals may affect decisions and actions taken by other entities near the autonomous vehicle 102, thereby affecting the predicted movement of other objects.

[0027] In some examples, the perception component 110 may receive sensor data from the sensors 104 and determine data associated with objects near the vehicle 102, such as static and / or dynamic data. The illustrated example of environmental data 134 including static and / or dynamic data is a top-down representation of such data, although any representation of static and / or dynamic data is contemplated, such as heat maps, object classifications associated with detected objects, instance segmentations, semantic segmentations, two-dimensional and / or three-dimensional bounding boxes, tracks, etc. The perception component 110 may additionally or alternatively determine route data specifying a destination of the vehicle, overall map data identifying road characteristics (e.g., features detectable by various sensor modalities that are useful for locating an autonomous vehicle), local map data identifying features detected near the vehicle (e.g., the location and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable by various sensor modalities), track data (e.g., the environmental representations, object detections, and / or tracks discussed herein), and / or the like.

[0028] In some examples, the perception component 110 may receive sensor data from the sensors 104 and determine data associated with objects near the vehicle 102, such as static and / or dynamic data. The illustrated example of environmental data 134 including static and / or dynamic data is a top-down representation of such data, although any representation of static and / or dynamic data is contemplated, such as heat maps, object classifications associated with detected objects, instance segmentations, semantic segmentations, two-dimensional and / or three-dimensional bounding boxes, tracks, etc. The perception component 110 may additionally or alternatively determine route data specifying a destination of the vehicle, overall map data identifying road characteristics (e.g., features detectable by various sensor modalities that are useful for locating an autonomous vehicle), local map data identifying features detected near the vehicle (e.g., the location and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable by various sensor modalities), track data (e.g., the environmental representations, object detections, and / or tracks discussed herein), and / or the like.

[0029] In some examples, perception component 110 may include a hardware and / or software pipeline, which may include one or more GPUs, ML models, Kalman filters, etc. In some examples, perception component 110 may monitor as much of the environment surrounding the autonomous vehicle as possible, which may be limited by sensor capabilities, object and / or environmental occlusion (e.g., buildings, elevation changes, objects in front of other objects), and / or environmental effects such as fog, snow, etc. For example, sensor data may include RADAR data, which perception component 110 may receive as input. Perception component 110 may be configured to detect as many objects and information about the environment as possible to avoid failing to consider events or object behaviors that should be taken into account by planning component 112 in determining trajectory 118.

[0030] The object classification may be part of the environmental data 134 and may be determined by the perception component 110 and may distinguish between various object types, such as, for example, a passenger vehicle, a pedestrian, a bicyclist, a delivery truck, a semi-truck, a traffic sign, and the like. The track may include past, current, and / or predicted object position, speed, acceleration, and / or direction. In some examples, the perception component 110 may include a prediction component that may determine a predicted portion of the track, which may include a predicted position, direction, steering speed, speed, acceleration, classification (with respect to a susceptible classification, such as a cyclist that may be a pedestrian), and the like. The prediction component may include a simulation component, a machine learning model, or in additional or alternative examples, the prediction component may include a Kalman filter, and the like. The perception component 110 may be used to determine the environmental state indicated by the prediction node as part of the tree search discussed herein. For example, the environmental state may include current and / or predicted environmental data 134.

[0031] The data generated by the perception component 110 may be collectively referred to as sensory data. As / when the perception component 110 generates the sensory data, the perception component 110 may provide the sensory data to the planning component 112 and / or the guidance system 114. In some examples, the sensory data may include the output of a sensor-specific pipeline (e.g., vision, LIDAR, RADAR) and / or a hybrid sensor pipeline (e.g., vision-LIDAR, RADAR-LIDAR).

[0032] The planning component 112 may use the sensory data received from the perception component 110 and / or the path received from the guidance system 114 to determine one or more trajectories, control the movement of the vehicle 102 to move around a path or route, and / or otherwise control the operation of the vehicle 102, although any such operation may be performed in a variety of other components (e.g., localization may be performed by a localization component, which may be based at least in part on the sensory data). For example, the planning component 112 may determine a route for the vehicle 102 from a first location to a second location, determine a smooth trajectory from a rough trajectory received from the guidance system 114, substantially simultaneously generate multiple potential trajectories for controlling the movement of the vehicle 102 according to a receding horizon technique (e.g., 1 microsecond, 0.5 seconds, 2 seconds, 5 seconds, 10 seconds, or any other short-term time period) to control the vehicle to follow the route (e.g., to avoid any of the detected objects) based at least in part on the path and sensory data and / or simulated sensory data (which may further include predictions regarding detected objects in such data), and select one of the potential trajectories as the trajectory 118 of the vehicle 102 that may be used to generate drive control signals that may be transmitted to the controller 116 to operate drive components of the vehicle 102. FIG. 1 shows an example of such a trajectory 118, represented as arrows indicating a target direction, a target steering angle, a target steering velocity, a target position, a target velocity, and / or a target acceleration for the controller to follow, but the trajectory itself may include instructions for the controller, which in turn may operate the drive system of the vehicle 102.

[0033] In some examples, the guidance system 114 may be a secondary guidance system to the nominal guidance system of the planning component 112. In some examples, the planning component 112 may generate one or more candidate actions for a tree search as discussed herein that are associated with an action node. The planning component 112 may receive a trajectory or path from the nominal guidance system and may receive a path generated by the guidance system 114 as discussed herein. The path received from the guidance system 114 may be treated as a contingent path by the planning component 112, for example, the planning component 112 may rely on the output of the nominal guidance system of the planning component 112 unless or until the output of the nominal guidance system causes the vehicle 102 to stop, or causes the vehicle 102 to invoke remote operations, or fails to output a feasible, collision-free, and / or economical path or trajectory. In such cases, the planning component 112 may switch to using the path output by the guidance system 114. In additional or alternative examples, the planning component 112 may generate a trajectory using a route output by the guidance system 114, across all time, in a particular region (e.g., in an area without lane references), based at least in part on sensor data and / or perception data, etc. In some examples, the guidance system 114 may perform a tree search as discussed herein, which may include managing the generation of action nodes (e.g., by sending requests to an application programming interface (API) associated with the planning component 112 and receiving candidate actions in response thereto) and / or managing the generation of prediction nodes (e.g., by sending requests to an API associated with a prediction component of the perception component 110 and receiving current and / or predicted environmental state data), etc.

[0034] The guidance system 114 may include one or more GPUs or may be communicatively coupled to one or more GPUs (e.g., via a publish / subscribe messaging system, via a data bus), and while the techniques discussed herein may be parallelized and spread across the threads of the GPUs, it is contemplated that the techniques discussed herein may include at least portions that are serial.

[0035] Exemplary System FIG. 2 illustrates a block diagram of an example system 200 that implements the techniques discussed herein. In some cases, the example system 200 may include a vehicle 202, which may correspond to the vehicle 102 in FIG. 1. In some cases, the vehicle 202 may be an autonomous vehicle configured to operate according to the Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the entire trip without the driver (or passenger) being expected to control the vehicle at any time. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having any other level or classification. Moreover, in some cases, the techniques described herein may be usable by non-autonomous vehicles as well.

[0036] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. Vehicle computing device 204 may correspond to computing device 106, and sensor 206 may correspond to sensor 104. System 200 may additionally or alternatively include computing device 214.

[0037] In some cases, the sensors 206 may correspond to the sensors 104 and may include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., Red-Green-Blue (RGB), Infrared (IR), intensity, depth, time-of-flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors, etc.), etc. The sensors 206 may include multiple instances of each of these or other types of sensors. For example, the RADAR sensors may include individual RADAR sensors located at the corners, front, rear, sides, and / or top of the vehicle 202. As another example, the cameras may include multiple cameras located at various locations around the exterior and / or interior of the vehicle 202. The sensors 206 may provide input to the vehicle computing device 204 and / or to the computing device 214. The positions associated with the simulated sensors discussed herein may correspond to the position and / or origin (e.g., focal point) of the sensor's field of view relative to the vehicle 202 and / or the direction of motion of the vehicle 202.

[0038] The vehicle 202 may also include emitters 208 for emitting light and / or sound, as described above. The emitters 208 in this example may include internal audio and visual emitters for communicating with occupants of the vehicle 202. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example may also include external emitters. By way of example and not limitation, the external emitters in this example may include lights or other indicators of vehicle actions (e.g., indicator lights, signs, light arrays, etc.) to signal direction of travel, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0039] Vehicle 202 may also include a network interface 210 that enables communication between vehicle 202 and one or more other local or remote computing devices. For example, network interface 210 may facilitate communication with other local computing devices and / or drive components 212 on vehicle 202. Network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Network interface 210 may additionally or alternatively enable vehicle 202 to communicate with computing device 214. In some examples, computing device 214 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0040] The network interface 210 may include physical and / or logical interfaces for connecting the vehicle computing device 204 to another computing device or network, such as the network 216. For example, the network interface 210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices. In some cases, the vehicle computing device 204 and / or the sensors 206 may transmit sensor data over the network 216 to the computing device 214 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0041] In some cases, the vehicle 202 may include one or more drive components 212. In some cases, the vehicle 202 may have a single drive component 212. In some cases, the drive component 212 may include one or more sensors for detecting conditions surrounding the drive component 212 and / or the vehicle 202. By way of example and not limitation, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive component, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive component, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects in the surroundings of the drive component, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap with or complement corresponding systems (e.g., sensors 206) of the vehicle 202.

[0042] The drive components 212 may include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, an on-board charging system, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). In addition, the drive components 212 may include a drive components controller, which may receive and pre-process data from sensors to control the operation of various vehicle systems. In some cases, the drive components controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functionality of the drive components 212. In addition, the drive components 212 may also include one or more communication connections that enable each drive component to communicate with one or more other local or remote computing devices.

[0043] The vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to the one or more processors 218. The memory 220 may correspond to the memory 108. The computing device 214 may also include a processor 222, and / or a memory 224. The processor 218 and / or 222 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor 218 and / or 222 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or any other device or portion of a device that processes electronic data and converts the electronic data into other electronic data that may be stored in registers and / or memory.

[0044] The memory 220 and / or 224 may be an example of a non-transitory computer-readable medium. The memory 220 and / or 224 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying figures are merely examples relevant to the discussion herein.

[0045] In some cases, memory 220 and / or memory 224 may store a localization component 226, a perception component 228, a planning component 230, a guidance component 232, a simulation component 234, an agent filter 236, and / or a system controller 238, any one of which zero or more portions may be hardware, such as a GPU, a CPU, and / or other processing unit. The perception component 228 may correspond to the perception component 110, the planning component 230 may correspond to the planning component 112, and the guidance component 232 may correspond to the guidance system 114.

[0046] In at least one example, the localization component 226 may include hardware and / or software for receiving data from the sensors 206 to determine the position, velocity, and / or orientation (e.g., one or more of x, y, z position, roll, pitch, or yaw) of the vehicle 202. For example, the localization component 226 may include and / or request / receive a map of the environment and be able to continually determine the position, velocity, and / or orientation of the autonomous vehicle within that map. In some cases, the localization component 226 may receive image data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, etc., utilizing simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position, attitude, and / or velocity of the autonomous vehicle. In some cases, the localization component 226 may provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for purposes of generating a trajectory and / or for purposes of generating map data. In some examples, the localization component 226 may provide the position and / or orientation of the vehicle 202 relative to the environment and / or sensor data associated therewith to the perception component 228.

[0047] In some cases, the perception component 228 may include a primary perception system and / or a prediction system implemented in hardware and / or software. For example, the prediction component may be the simulation component 234, although in additional or alternative examples, the prediction component and the simulation component 234 may be separate components. Regardless, the perception component 228 (as well as its prediction component and / or simulation component 234) may cooperate with the guidance component 232 to determine the environmental conditions discussed herein. The perception component 228 may detect objects in the environment surrounding the vehicle 202 (e.g., identify that an object is present), classify the objects (e.g., determine an object type associated with the detected object), segment the sensor data and / or other representation of the environment (e.g., identify a portion of the sensor data and / or representation of the environment as associated with the detected object and / or object type), determine characteristics associated with the object (e.g., a track identifying a current, predicted, and / or previous position, direction, speed, and / or acceleration associated with the object), and so forth. The perception component 228 may include a prediction component that predicts the actions / states of dynamic components of the environment, such as moving objects. In some examples, the perception component 228 may include a simulation component 234 as part of the prediction portion of the perception component 228. The data determined by the perception component 228 is referred to as sensory data. Although the agent filter 236 is shown as being part of the simulation component 234, in additional or alternative examples, the perception component 228 may include an agent filter 236.

[0048] The planning component 230 may receive the position and / or orientation of the vehicle 202 from the localization component 226 and / or the sensory data from the perception component 228, and may determine instructions for controlling the operation of the vehicle 202 based at least in part on any of this data. In some examples, determining the instructions may include determining the instructions based at least in part on a format associated with a system with which the instructions are associated (e.g., a first instruction for controlling the movement of the autonomous vehicle may be formatted in a first format of messages and / or signals that the drive component 212 can parse / have executed (e.g., analog, digital, pneumatic, kinematic, as may be generated by a system controller of the drive component 212), and a second instruction for the emitter 208 may be formatted according to a second format associated therewith). In some examples, if the planning component 230 may include hardware / software in-the-loop in a simulation (e.g., for testing and / or training the planning component 230), the planning component 230 may generate instructions that may be used to control the simulated vehicle. These instructions may additionally or alternatively be used to control the movement of a real-world version of the vehicle 202, for example, when performing a simulation run on the vehicle while the vehicle 202 is in operation.

[0049] While the planning component 230 may be a primary component for determining control instructions for the vehicle 202, such as during operation of the vehicle 202 in nominal conditions, the planning component 230 may further include the guidance component 232, and / or the vehicle 202 may additionally include the guidance component 232, separate from the planning component 230. The guidance component 232 may determine a trajectory and / or path for controlling the vehicle simultaneously with the planning component 230, such as to determine an accidental trajectory and / or path for controlling the vehicle 202 when the trajectory determined by the planning component 230 fails to be generated (e.g., the planning component 230 is not capable of determining a proper trajectory that avoids an object) and / or it violates a comfort metric, such as a threshold acceleration and / or jerk, or a rule of the road. The guidance component 232 may perform the tree search discussed herein and manage determining the action node and / or prediction node of the tree search by transmitting a request for the planning component to generate a candidate action based at least in part on the determined environment in relation to the prediction node. The guidance component 232 may receive an initial state of the environment from the perception component 228 (i.e., relative to a root node of the tree search), and the guidance component 232 may transmit this initial environmental state to the planning component 230 and may receive one or more candidate actions from the planning component 230. The guidance component 232 may transmit at least one of these one or more candidate actions to the simulation component 234 and / or the prediction component of the perception component 228, which may determine a predicted state of the environment based at least in part on the candidate action. This process may be repeated until a time horizon, distance, progress, target location, and / or suitable path along the route is reached / determined.

[0050] For example, the time horizon may be a length of time from the current time into the future (e.g., 500 ms, 1 s, 2 s, 5 s, 8 s, 10 s). This length of time may be associated with controlling the vehicle for the next m units of time, where m is a positive integer. The distance may define a total distance covered by the constituent actions constituting the path, while the progress along the route may be a displacement along / relative to the route. In additional or alternative examples, a goal location may be used to terminate the tree search. For example, upon determining a path to reach a goal location in the environment, the tree search may output the path and terminate. In additional or alternative examples where the guidance system is used when the nominal planning component fails to create a valid trajectory or path, the guidance system may terminate upon determining a valid path (e.g., a path that is collision-free and complies with a rule set, which may specify a comfort metric, compliance with laws, etc.). In additional examples, the iterations may continue until a goal is achieved (e.g., a successful lane change, a successful merge, or any other completed action). In any one or more examples, any combination of the above may be further used as decision points for branching the tree.

[0051] The simulation component 234 may operate on the vehicle 202 and / or on the computing device 214. The simulation component 234 may determine a simulation of the environment and / or the vehicle 202, such as simulating the performance of candidate actions by the vehicle 202, and a predicted state of the environment based at least in part on the passage of time and in response to the performance of the candidate actions by the vehicle 202. For example, the simulation may include representations of positions, orientations, movements, and / or qualities of parts of the environment and / or the vehicle 202. The environment may include agents such as another vehicle, pedestrians, vegetation, buildings, signs, etc.

[0052] The simulation component 234 may receive candidate actions and environmental states (which may be current environmental states determined by the perception component 228 or predicted environmental states determined by the prediction component of the perception component 228 or by the simulation component 234) from the guidance component 232 to determine simulation data, which may be a two-dimensional or three-dimensional representation of the scenario. The simulation data may be used to instantiate and run the simulation. The candidate actions may be used to control the simulated movement of the vehicle 202 during the execution of the simulation. The three-dimensional representation may include position, orientation, geometric data (e.g., polygonal representations, digital wire mesh representations), and / or motion data associated with one or more objects of the environment, and / or may include material, lighting, and / or illumination data, although in other examples this data may be omitted. In additional or alternative examples, the simulation component 234 may include computational structures (e.g., algorithmic and / or mathematical representations used by a computing device in performing the described operations that are not intended to be (and / or cannot be) visualized).

[0053] The scenario data may include a two-dimensional representation of the environment associated with the scenario, the objects contained therein, and the properties associated therewith, all of which may be part of the scenario associated with the log data. For example, the scenario data may identify the location of the object, the area occupied by the object, the speed and / or acceleration associated with the object, whether the object is static or dynamic, the object type associated with the object (e.g., classifications such as "pedestrian," "bicyclist," "vehicle," "large vehicle," "traffic light," "traffic sign," "building," "road," "crosswalk," "sidewalk," etc.), and / or other kinematic properties associated with the object and / or object type (e.g., coefficient of friction, elasticity, malleability). With respect to the environment itself, the scenario data may identify the topology of the environment, weather conditions associated with the environment, lighting conditions (e.g., sunny, cloudy, night), the location of light sources, etc. In some examples, the topology associated with the scenario data, the locations and dimensions of fixed objects (e.g., buildings, trees, signs), etc. may be generated based at least in part on the map. In some examples, the scenario data may be used (e.g., by the simulation component 234) to instantiate three-dimensional representations of objects and / or a simulated environment may be instantiated based at least in part on map data (e.g., which may define the topology of the environment, the locations and / or dimensions of fixed objects such as signs, plants, and / or buildings) and / or the scenario data.

[0054] Additionally or alternatively, the simulation may include simulated objects controlled by an agent behavior model, discussed in more detail in U.S. Patent Application Publication No. 2020 / 0133991, filed June 1, 2020, in addition to or in lieu of the nominal prediction component of the simulation component or the prediction component of the perception component 110, the entirety of which is incorporated herein by reference. The agent behavior model may control the simulated movement of a simulated representation of a dynamic object, such as a reactive dynamic object. In some examples, the simulation may be performed as part of a forecast / prediction operation, such that one or more simulations may be performed to determine expected scenarios (e.g., predicted environmental state data) based on candidate actions generated according to the tree search discussed herein.

[0055] In some examples, the simulated sensor may determine simulated sensor data based at least in part on a simulation performed by the simulation component 234. For example, U.S. Patent Application Publication No. 2012 / 0133991 filed on September 24, 2019, which is incorporated herein by reference in its entirety, discusses this in further detail. In additional or alternative examples, the simulation performed by the simulation component may itself include simulated sensor data. The perception component 228 (e.g., a copy of which may include software and / or hardware, which may include hardware-in-the-loop simulation) may receive such sensor data and / or output the simulated sensor data that is provided as input to the planning component 230. The planning component may use the perception data to determine instructions for controlling the movement of the vehicle 202, which may be used to control at least a simulated representation of the vehicle 202 in the simulation, and in some examples, such as in examples where the simulation component 234 executes on the vehicle during real-world operation, may additionally be used to control the real-world movement of the vehicle 202.

[0056] In some examples, the simulation component 234 may additionally or alternatively store a rule set and determine whether a candidate action passed or failed a scenario based at least in part on the rule set. The rule set may be associated with and / or include a motion constraint. In some examples, the rule set may be part of or replaced by an event detection system (US Pat. No. 6,399,433, filed Nov. 13, 2019, the entire contents of which are incorporated herein by reference) and / or a crash monitoring system (US Pat. No. 6,399,433, filed Dec. 4, 2019).

[0057] The simulation component 234 may include an agent filter 236, which may include a machine learning model that is trained to classify dynamic objects detected by the perception component 228 as reactive or inactive. In at least one example, the classification need not be binary, but may be a score, such as a number between 0 and 1, where 0 indicates that the object is inactive and 1 indicates that the object is reactive. In some examples, the agent filter 236 may receive as input the track associated with the dynamic object and the candidate actions of the vehicle, and may determine whether the dynamic object is inactive or reactive according to a binary classification or regression, as described above. A purely inactive agent will never change its behavior based on the behavior / candidate actions of the vehicle 202, while the actions taken by the reactive object are very likely to be influenced by the actions of the vehicle 202. In additional or alternative examples, the agent filter 236 may determine the likelihood (e.g., covariance, posterior probability) that the dynamic object will modify its associated action based on the actions of the vehicle 202. A dynamic object associated with a probability that meets or exceeds a threshold probability may be classified as being reactive, while another dynamic object associated with a probability that does not meet the threshold may be classified as an inactive object.

[0058] In some examples, training the agent filter 236 may include receiving log data from the vehicle, which may include perception data and trajectories implemented by the vehicle. The perception data may identify data related to the dynamic object, such as the dynamic object and / or a track associated with the dynamic object. To identify a label that the dynamic object was responsive to the vehicle's actions, or to generate a score that identifies such, the computing device may determine a correlation score based on changes over time in the track of the dynamic object relative to the trajectory implemented by the vehicle. In some examples, a filter may be applied to dynamic objects in the log data to limit dynamic objects that are unlikely to have modified their behavior based on the vehicle's actions. For example, the filter may remove dynamic objects that are located more than a threshold distance away from the vehicle based on (the dynamic object's speed, in some cases) the dynamic object's direction, etc.

[0059] The predicted state of the environment may be determined by the prediction component of the perception component 228 and / or the simulation component 234 based at least in part on whether any dynamic objects in the environment are classified as reactive. The predicted state of reactive objects may be simulated by the simulation component 234, while the predicted state of inert objects may be determined by the prediction component of the perception component 228, such as a Kalman filter for such prediction, or other simple propagation of the current state of the inert objects (e.g., dead reckoning). The inert object prediction component may use fewer calculations, while the dynamic object prediction component (e.g., the simulation component 234) may require more calculations in some cases.

[0060] Memory 220 and / or 224 may additionally or alternatively store a mapping system, a planning system, a ride management system, etc. Although perception component 228, planning component 230, and / or simulation component 234 are shown as being stored in memory 220 and / or 224, perception component 228, planning component 230, guidance component 232, simulation component 234, and / or agent filter 236 may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware.

[0061] As described herein, the localization component 226, the perception component 228, the planning component 230, the guidance component 232, the simulation component 234, the agent filter 236, and / or other components of the system 200 may include one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, the guidance component 232, the simulation component 234, and / or the agent filter 236 may each include a separate ML model pipeline. In some examples, the ML model may include a neural network. An exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each of the layers in a neural network may include another neural network or may include any number of layers (whether convolutional or not). As may be understood in the context of the present disclosure, the neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0062] Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms can include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative bisection 3D modeling, etc.), and the like. (ID3), Chi-squared Automatic Interaction Detection (CHAID), Decision Stump, Conditional Decision Trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average One Dependence Estimators (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), Clustering algorithms (e.g., k-means, k-median, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning algorithms (e.g., Pairwise Clustering, The learning algorithms may include, but are not limited to, deep learning algorithms (e.g., deep Boltzmann machines (DBM), deep belief networks (DBN), convolutional neural networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBM), gradient boosted regression trees (GBRT), random forests), SVM (support vector machines), supervised learning, unsupervised learning, semi-supervised learning, and the like.Further examples of architectures include neural networks such as ResNet-50, ResNet-101, VGG, DenseNet, PointNet, etc. In some examples, the ML models discussed herein may include PointPillars, SECOND, top-down feature layers (see, e.g., U.S. Patent No. 6,399,323, the entirety of which is incorporated herein by reference), and / or VoxelNet. Latency-optimized architectures may include MobilenetV2, Shufflenet, Channelnet, Peleenet, etc. The ML models may include residual blocks such as Pixor, in some examples.

[0063] Memory 220 may additionally or alternatively store one or more system controllers (which may be part of the drive component) that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers may communicate with and / or control corresponding systems of drive component 212 and / or other components of vehicle 202. For example, planning component 230 may generate instructions based at least in part on sensory data generated by perception component 228 and / or simulated sensory data and transmit the instructions to the system controller, which may control operation of vehicle 202 based at least in part on the instructions.

[0064] 2 is shown as a distributed system, it should be noted that in alternative examples, components of the vehicle 202 may be associated with the computing device 214 and / or components of the computing device 214 may be associated with the vehicle 202. That is, the vehicle 202 may perform one or more of the functions associated with the computing device 214, and vice versa.

[0065] Exemplary Path Generation Using Tree Search and / or Agent Filters 3A-3C show diagrammatic flow diagrams of an example process 300 for generating a path for controlling a vehicle (e.g., vehicle 202) using an agent filter and / or a tree search with alternating estimated costs that may be determined at least in part based on sensor data and / or perception data, and may include an occupancy map showing static objects and / or a map of probabilities of predicted positions and / or velocities of dynamic objects. In some examples, the example process 300 may be performed by a guidance component (e.g., guidance component 232) of an autonomous vehicle, although in at least some examples, the example process 300 may additionally or alternatively be performed by a planning component, a simulation component, a perception component, and / or a prediction component of an autonomous vehicle. In the latter examples, the guidance component may coordinate the operation of the various components, such as by transmitting API requests to APIs associated with each of the components and using the API responses to perform the tree search discussed herein. In another example, the guidance component may coordinate the operation of the components using messages transmitted to / from the various components via a publish-subscribe network. In some examples, the tree search performed by the guidance component may include performing an improved Monte Carlo Tree Search (MCTS) using techniques discussed herein, including agent filtering, upper / lower bound cost estimation, and / or defaulting to a default policy, partially observable Monte-Carlo planning (POMCP), a Markov Decision Process (MDP), such as a partially observable MDP (POMDP), or the like.

[0066] At operation 302, the example process 300 may include receiving a route associated with at least a start location in an environment, according to any of the techniques discussed herein. Figure 3A illustrates an environment 304 in which a vehicle 306 performing the example process 300 is located. The start location may be associated with a current location of the vehicle 306, and the route may specify an end location and, in some examples, may include intervening goals or maneuvers, such as exiting a highway, trying to stay in a particular lane, aiming to park on a particular block (though not a particular location, although in some examples a particular portion of a block may be identified), etc.

[0067] At operation 308, the example process 300 may include receiving sensor data from one or more sensors according to any of the techniques discussed herein. The sensors may be associated with the vehicle and / or another computing device. Operation 308 may additionally or alternatively include determining environmental state data based at least in part on the sensor data. In some examples, the perception component may determine environmental state data 310 (abbreviated in the figure as “state 310” for space reasons), and for any dynamic objects detected by the perception component, the agent filter may determine whether the dynamic object is reactive or inactive (or a score associated therewith, as described above). The environmental state data 310 may be associated with the most recently received set of sensor data (e.g., the current time, although there may be a small delay between receiving the sensor data and determining the perception data).

[0068] Further indicating, the environmental state data 310 may include a position, orientation, and / or characteristics of the vehicle 306 in the environment, and the environmental state data 310 may correspond to real-time operation of the autonomous vehicle. The environmental state data 310 may additionally or alternatively include an indication of an object type (e.g., car 312, large vehicle 314, car 316, building 318, building 320) associated with one or more objects and / or characteristics (e.g., position, speed, acceleration, direction, material type, kinematic coefficients) associated with one or more objects. Note that while the environmental state data 310 is represented as a two-dimensional image, in additional or alternative examples, the environmental state data 310 may include data structures such as pubsub messages, three-dimensional representations, and the like. In some examples, the environmental state data 310 may further include a prediction of whether an occluded object is present, or the like, as discussed in further detail in U.S. Patent Application Publication No. 2010 / 0133994, filed May 8, 2019, the entirety of which is incorporated herein by reference. In an additional or alternative example, the prediction of whether an occluded object is present may be determined by a machine learning model that receives environmental state data as input and outputs a field of probabilities. Any area of ​​the environment associated with a probability that meets or exceeds a threshold may be output as a potential false negative, which may be used as part of candidate action generation.

[0069] The environmental state data may include objects classified as dynamic by the perception component. For example, dynamic objects may be referred to herein as agents and may include vehicles, bicyclists, pedestrians, balls, plastic bags blowing in the wind, and / or any other movable objects or objects likely to move within a time period. Objects such as benches or tables may be movable but unlikely to move and may be considered static objects in a time period related to the operation of the vehicle. The environmental state data 310 may include dynamic objects and may include dynamic object classifications and / or probabilities determined by the agent filter in association with the dynamic objects. For example, the classifications may include whether and / or the likelihood that the dynamic object is inactive or reactive. As described above, the agent filter may include an ML model that has been trained to receive object tracks associated with the dynamic objects, the current state of the vehicle and / or candidate actions discussed further herein, and / or sensor data associated with the dynamic objects, and to determine a classification and / or confidence score (e.g., posterior probability, likelihood) that the dynamic object is inactive or reactive by neural networks or any of the other ML techniques described above. In some examples, if the confidence score determined by the ML model meets or exceeds a confidence threshold, the detected object may be classified as a reactive object, otherwise the detected object may be classified as an inert object. In yet another example, the ML model may additionally or alternatively output the classification itself from the last layer in addition to or instead of the confidence score.

[0070] Inactive objects may be dynamic objects that are unlikely to change their behavior (e.g., have a reliability score below a reliability score threshold) and / or that depend at least in part on the actions of vehicle 306, while reactive objects may be dynamic objects that are likely to change their behavior based on the actions of vehicle 306. Inactive objects are shown using cross-hatching, indicated at 322 in the figure. Reactive objects are shown using cross-hatching, indicated at 324 in the figure. Static objects are shown using cross-hatching, indicated at 326 in the figure. Note that in FIG. 3A, traffic is moving to the right, and thus large vehicle 314, for example, has already substantially passed vehicle 306, and thus may be unlikely to change the action taken by it based on the actions of vehicle 306. Vehicles 312 and 316, on the other hand, have not yet passed vehicle 306, and may be significantly more likely to base their actions, at least in part, on the actions of vehicle 306.

[0071] 3B, in operation 328, the example process 300 may include determining a root node 330 of a tree search based at least in part on the sensor data according to any of the techniques discussed herein. In some examples, determining the root node may include determining a data structure 332 for the tree search, which may include setting up and storing a directed acyclic graph (DAG), upper confidence bounds applied to trees (UCT), determinized sparse partially observable tree (DESPOT), etc., to model the control and environmental states. The root node may be associated with the current time and / or the most recent sensor data or batch of sensor data. Thus, the root node may be associated with sensory data, which may or may not include predictive data. In other words, the root node may identify environmental state data including current positions, orientations, velocities, accelerations, classifications, etc. of static and / or dynamic objects in the environment (including similar information about the vehicle that may be generated by the vehicle's localization components), and may additionally or alternatively include historical data of the same.

[0072] Predictions of how the object will behave in the future and correspondingly how this data will change in the future may be associated with prediction nodes as discussed herein, and in some examples, the prediction data for the current time step may be associated with a root node. In other words, the root node may include prediction data identifying one or more possible future states of the environment, which may include the current state of the environment including the object therein, localization data associated with the vehicle (e.g., determined by SLAM), and / or the position, orientation, velocity, acceleration, classification, etc. of the object associated with the future time.

[0073] The diagram illustrates prediction nodes (and root nodes, which may be prediction nodes) as squares and action nodes as circles. The dashed lines and circles 334 represent relationships between the root node 330 and as yet undiscovered / determined action nodes based on the root node 330. The root node 330 may identify environmental state data 310 and one or more predicted environmental scenarios. For simplicity and space reasons, only current environmental state data is displayed in FIGS. 3A-3C, but predicted environmental state data may be further associated with the root node 330, such as the predicted environmental state shown in FIGS. 6A and 6B.

[0074] In operation 336, the example process 300 may include determining a first candidate action for controlling the movement of the vehicle (based at least in part on the previous prediction node) according to any of the techniques discussed herein. The candidate action determined in operation 336 may be determined at least in part on the prediction node of the most recently determined layer of prediction nodes. For example, FIG. 3B shows only the first layer of prediction nodes, which only includes the root node 330. FIG. 3C shows the second layer of prediction nodes, which includes prediction nodes 350 and 352. Determining the first candidate action may include providing the planning component with environmental state data associated with the prediction node on which the candidate action is based. For example, the first action node 338 may be indicated with one or more candidate actions based on the environmental state data indicated by the root node 330. FIG. 3B shows one such candidate action, candidate action 340, which includes controlling the vehicle to move straight forward.

[0075] The environmental state data may be current environmental state data (if the predictive node is a root node) or associated predicted environmental state data, as described above. Regardless, determining the first candidate action in the planning component may include a nominal method of trajectory planning. In an additional or alternative example, determining the candidate action based at least in part on the environmental data may include a trajectory determination system separate from the nominal trajectory generation system of the planning component. This separate system may determine the candidate action based at least in part on lane reference type, target type, extension variable, offset, multiplier, and / or trend type. This is also discussed in more detail in FIG. 8A and FIG. 8B.

[0076] The lane criterion type may be an indication of whether the lane criterion for generating the candidate action should be generated using sensor data or using predefined lane criterion, such as may be indicated in a pre-generated map. The lane criterion is shown in FIG. 8 and may or may not be associated with the center of the lane (e.g., the lane criterion may be the center of the lane for a straight lane portion, but for a curve, the lane criterion may be biased to the inside or outside of the curve).

[0077] The goal type may define the type of action to accomplish the current route or mission. For example, the goal type may specify the vehicle's current lane, an adjacent lane, a parking space, a location in an open space (e.g., where no lane markings exist), etc.

[0078] The expansion variables may identify weights, distances, coefficients, and / or other limits on how far unoccupied space can be explored horizontally (and / or vertically in some examples) (e.g., how far a candidate action can move a vehicle laterally). For example, the expansion variables can be general constraints on how far candidate actions can differ.

[0079] The offset may identify a predetermined distance from the lane reference to use to iterate the search for candidate actions. The distance may additionally or alternatively be dynamically determined based at least in part on sensor data such as the speed of the vehicle, the complexity of the environment (see U.S. Patent Application Publication No. 2021 / 0133634, filed February 24, 2021, which is incorporated by reference in its entirety), etc.

[0080] The multiplier may be a factor between 0 and 1 that may be multiplied by the current maximum speed allowed by law to determine a maximum speed associated with the candidate action. The multiplier may be randomized, may be varied according to a pattern, and / or may be constrained based at least in part on a range set by a planning component based at least in part on environmental condition data and the vehicle's previous trajectory.

[0081] The propensity types may be associated with higher curvature, speed, and / or acceleration, and may identify curvature, speed, and / or acceleration constraints associated with various behavior types, such as "assertive," which may be required if the perception component detects a complex environment or other aggressive traffic, "nominal," which may provide a baseline for typical interactions with other agents, "conservative," and / or "submissive." As discussed in further detail in U.S. Patent Application Publication No. 2010 / 0139994, filed August 28, 2020, the entirety of which is incorporated herein by reference, the perception engine and / or planning component may work together to determine the propensity type to be used.

[0082] Once the planning component generates a first candidate action, the guidance component may update the data structure 332 to include a first action node 338 that identifies the first candidate action. Figure 3B also shows two more action nodes 342 and 344, which are shown in dashed lines because they may not be generated in cases where the tree search algorithm finds a low-cost path with minimal exploration. In other words, action nodes 342 and 344 may not yet be explored as of now, but may be generated upon further iterations of operation 336 to enumerate additional candidate actions.

[0083] In some examples, a first candidate action may be associated with controlling a vehicle for a first time period. As discussed below, a candidate action of a layer deeper than the layer associated with the first candidate action (e.g., which includes action nodes 342 and 344) may be associated with controlling a vehicle for a second time period. In some examples, the time periods associated with each subsequent layer of action nodes may be equal, or in additional or alternative examples, the time periods may increase in length (e.g., exponentially, logarithmically). For example, a first candidate action may be associated with controlling a vehicle for a period of 1 second, a second candidate action associated with an action node one layer deeper than the first layer (e.g., action node 356) may control a vehicle for a period of 1.1 seconds, a third layer may control a vehicle for a period of 1.25 seconds, and so on. This increasing time period may ensure that greater precision and / or accuracy is achieved with respect to the impending action, while also ensuring that a more distant action does not control the vehicle in a manner that results in higher costs / negative consequences.

[0084] In operation 346, the example process 300 may include determining a first cost associated with the first candidate action using a lower bound cost according to any of the techniques discussed herein. In some examples, determining the first cost may be part of determining the first candidate action in operation 336, and / or the cost determination may occur simultaneously or upon receiving the first candidate action using a separate processing unit. In some examples, the guidance system may determine the cost, which may be based at least in part on the environmental condition data. In particular, the cost may be based at least in part on various sub-costs such as a proximity cost, a safety cost, a comfort cost, and / or a progress cost. These sub-costs may be based at least in part on the environmental condition data indicated by the last predicted node (whether the last predicted node is a root node or another predicted node). The proximity cost may be based at least in part on a minimum, average, or other distance that the candidate action will move the vehicle from a static and / or dynamic object. The safety cost may include a score indicating compliance with the rules of the road, proximity to other objects and / or speed associated with the candidate action (e.g., the safety cost may penalize candidate actions that are close to the object (e.g., within a threshold distance of the object) and moving at a high speed, may not penalize or may only give a small penalty to candidate actions that are close to the object but associated with a low speed, and high speed candidate actions that are far from other objects may not be penalized by this cost), and / or proximity to undriveable surfaces (e.g., sidewalks, buildings, closed lanes). In examples where the safety cost includes a variable cost based on speed and lateral distance to the object, the cost may be determined based at least in part on a hinge function, such as an L1 or L2 hinge function.In some examples, the hinge point in the hinge function where the penalty begins to be applied may be based on the distance to the object, the speed associated with the candidate action, the object track, and / or the object type. For example, the penalty may begin to be applied farther away from the motorcyclist than from the vehicle, and / or the penalty may be higher / severe for the motorcyclist than for the vehicle. Moreover, the higher the speed associated with the candidate action, the more severe the penalty may be once the candidate action is within a threshold distance (e.g., the hinge point of the hinge function) of the vehicle. In at least one example, the threshold distance for applying the penalty specified by the L1 or L2 hinge function may be based at least in part on the speed associated with the candidate action. In other words, a fast candidate action will be penalized farther away from the object than a slow candidate action, and the closer the fast candidate action is to the object, the more severe the L1 or L2 penalty may be compared to the same distance from the slow candidate action to the object (e.g., the steeper the slope in the L1 case, the higher the coefficient and / or squared value).

[0085] The comfort cost may be based at least in part on the speed, jerk, and / or acceleration associated with the candidate action and / or whether the candidate action would violate a threshold jerk and / or acceleration. The progress cost may be based at least in part on the completion of a mission or sub-goal (e.g., parking on a portion of a block, parking in a block, changing lanes) and / or the displacement of the vehicle along a route. For example, the progress cost may reward the further along the route the vehicle would be if the candidate action were performed. The cost calculated as the reward may have an opposite sign to the other sub-costs. For example, if there is a positive cost for a candidate action that would violate a comfort metric (e.g., the candidate action would exceed a threshold jerk), the reward may be a negative sub-cost. Further details regarding how to determine the cost are discussed in U.S. Patent Application Publication No. 2020 / 0133994, filed May 11, 2020, the entirety of which is incorporated herein by reference.

[0086] In at least one example, the cost associated with a particular action node may include the cost of arrival (e.g., for any action node deeper than the first layer, the sum of the costs of all action nodes leading to that action node), the cost of performing the action (e.g., this may include the costs discussed above, such as comfort cost, progress cost, etc.), and the cost of proceeding further after that action node (which may also be characterized as the cost of transitioning to a different state in the future). Modeling this future cost, also called cost-to-go, may be complex and require a large amount of computational power when the number of action nodes explored in the tree search is considered. In reinforcement learning, cost-to-go is also called the "value" of being in a particular state.

[0087] In some examples, instead of determining all of the sub-costs, operation 346 may include using a floor cost or ceiling cost to substitute for determining at least a portion of the sub-costs, such as a cost-to-go. For example, the floor cost may be 0 and the ceiling cost may be the cost of using a default action. The floor cost may be a predefined heuristic, but in additional or alternative examples, the floor may be determined by a machine learning model trained at least in part based on simulating or operating a vehicle and determining a minimum cost of an action taken by the vehicle for similar scenarios. The machine learning model may determine the floor cost based at least in part on environmental scenario data and / or a track associated with the vehicle (i.e., that data may be provided as an input). In yet another example, the floor cost may be updated after all or most of the candidate actions based on the prediction node have been determined. In such a case, the floor cost may be updated to be the cost of the candidate action with the lowest cost.

[0088] The default action may be to hold the current speed, direction, steering speed, etc., or the default action may be a predefined action, such as to go straight ahead at the speed limit detected as being associated with the portion of the road where the vehicle is located. In another example, the default action may be determined by a dedicated machine learning model for determining a default action based at least in part on the sensory data. Note that in the example shown in Figures 3A-3C, the floor cost is used in the first action layer after the root node. This is done only as an example, and a ceiling cost may be used instead of the floor cost. Figures 3A-3C only show that floor costs and ceiling costs may be used alternately. In some examples, the floor costs and ceiling costs may alternate according to a ratio of three floor costs for two ceiling costs, two ceiling costs for two floor costs, four floor costs for one ceiling cost, etc.

[0089] In some cases, a ramping ratio may be used to change the ratio of floor costs to upper bound costs used in successive layers. For example, the upper bound costs may be used more or exclusively in the lowest layers (e.g., the first two or three) before introducing the floor bound costs and increasing the frequency with which the floor bound costs are used for successive layers (or vice versa). In some instances where the tree is deep enough, the ramping ratio may reach a steady state when the floor bounds are used exclusively or when a particular ratio is used (e.g., even at a 1:1 ratio). Purely using the floor bound costs guarantees finding the optimal route because the floor bound costs force the tree search algorithm to explore more of the tree. However, by incorporating the upper bound costs, the tree search algorithm becomes more greedy and the tree search algorithm may be adjusted by balancing the ratio of use of the floor bound costs to use of the upper bound costs. In other words, adjusting the tree search algorithm may include balancing the algorithm between the completeness of the amount of space explored / time and amount of computation to find a route and finding the best route.

[0090] In some examples, search parameters such as the ratio of floor costs to ceiling costs used, or whether floor costs or ceiling costs are used exclusively, may be determined at least in part based on sensory data using a machine learning model. For example, training data may be generated by experimentally varying the ratio used, or using either floor or ceiling costs exclusively, and storing the generated paths, the time it took to compute the path, the computation cycles, and / or the number of nodes and / or layers, the cost associated with the path, and / or how the floor / ceiling parameters were set. Based on sensory data available to the machine learning model, such as environmental state data indicated by the root node, the machine learning model may be trained to output tree search parameters predicted to reduce the computation cycles used, the number of nodes explored, and / or the cost associated with the path. The parameters may additionally or alternatively include the depth of the tree search, the width of the tree search, sampling parameters (how to change predictions, number of predictions made, etc., discussed in further detail with reference to Figures 8A and 8B), parameters for determining whether to group predicted nodes into a single predicted node (e.g., whether an exact match of dependent candidate actions is required, a threshold distance used to identify what qualifies as "similar," and / or k-means clustering parameters), whether dynamic objects may be reclassified during the tree search, and / or how many layers the search may explore before reclassification, etc.

[0091] By replacing at least a portion of the costs with lower or upper bound cost estimates, it may not be necessary to run a simulation or prediction component before the first candidate action is output, and / or this replacement may reduce computational costs and increase the speed of the tree search algorithm.

[0092] Referring to FIG. 3C, at operation 348, the example process 300 may include determining a first predictive node associated with a future state of the environment based at least in part on the first candidate action according to any of the techniques discussed herein. In at least one example, determining the first predictive node may include executing a predictive component, which may execute a Kalman filter, a machine learning model, or a simulation. Running the simulation may include instantiating the simulation based at least in part on the environmental state data and the first candidate action. The instantiation may use a set of template models that may include representations of various objects, such as three different vehicle models, four different pedestrian models, etc., which may be scaled and positioned as necessary to match the environmental state data indicated by the predictive node on which the candidate action is based. There may be any number of different models. In some examples, the template model may include a three-dimensional model of the object's surface without any texturing, although in additional or alternative examples, the model may include texture. The template model may include a polygon mesh, a triangular mesh, etc. In some examples, models associated with dynamic objects may have a higher polygon count than models associated with static objects. In yet another example, objects in a simulation may be simplified to basic two-dimensional or three-dimensional geometric shapes corresponding to the dimensionality of the simulation. For example, vehicles and / or pedestrians may be represented as rectangles of various shapes, pedestrians may be represented as cylinders, and so on.

[0093] In some examples, the instantiated simulated environment may include lighting and / or textures, although the techniques described herein work without lighting and / or textures. For two-dimensional simulations, lighting and textures may not be used. Moreover, lighting and textures may not be used for three-dimensional simulations. The simulated environment may include a model of the vehicle 306 itself. In some examples, instantiating the environmental portion of the simulated environment may be based at least in part on map data and / or sensory data (which may be indicated by environmental state data associated with the predictive node on which the first candidate action was based), and the objects in the simulated environment may be based at least in part on environmental state data indicated by the predictive node on which the first candidate action was based.

[0094] Executing the simulation may include controlling a simulated representation of the vehicle 306 according to the candidate action and based on the instantiation of the simulation described above, which may include using the environmental state data indicated by the predictive node on which the candidate action was based. Executing the simulation may additionally or alternatively include simulating the movement of a dynamic object in the simulated environment. Simulating the movement of the dynamic object may depend on whether the dynamic object is indicated as reactive or inert, as determined by the agent filter, which may be indicated in the environmental state data associated with the predictive node on which the candidate action was based. If the agent filter classifies the dynamic object as inert, a Kalman filter or other simpler prediction method may be used to determine a predicted trajectory of the dynamic object. However, if the agent filter classifies the dynamic object as reactive, the simulation component may control the movement of the dynamic object using a separate instance of the planning component, an agent behavior model (see U.S. Patent No. 6,399,323), and / or one or more ML models trained to control the simulation of the dynamic object based at least in part on the candidate actions. For example, the component that controls the simulated movement of the dynamic object may generate commands for controlling the simulated dynamic object based at least in part on the simulated execution of the candidate actions by the simulated vehicle 306. However, in another example, the simulation may be the output of a mathematical construct or machine learning model.

[0095] Once the simulation is complete (e.g., upon completing the candidate actions), the resulting predicted tracks of the objects in the environment (e.g., positions, orientations, etc., as discussed above), including the resulting track of the vehicle 306 from performing the first candidate action, may be used to determine updated environmental state data. The data structure 332 may be updated to include a prediction node 350 indicating this updated environmental state data and a predicted state of the environment that may result from implementing the first candidate action. In some examples, the simulation may be re-run with slightly different variables (e.g., changing the tendency of the dynamic objects from “conservative” to “aggressive,” “compliant,” or “nominal”) to determine second updated environmental data associated with a different prediction node, prediction node 352. In some examples, the simulation component may output multiple potential scenarios, each of which may be associated with a likelihood. In such examples, the guidance component may create a prediction node for each potential (predicted) scenario associated with a likelihood of meeting or exceeding a likelihood threshold.

[0096] In another example, one or more predictions may be grouped together at a single node based at least in part on determining the similarity between the predictions and / or determining the similarity of the candidate actions determined based on those predictions. In other words, after the next set of candidate actions is determined at a subsequent action node layer (e.g., the layer including the second action node 356), the process may include determining the similarity between the candidate actions of that layer. In some examples, this may include determining the Euclidean distance between the locations in space and / or space-time defined by the two candidate actions (e.g., the Euclidean distance at a particular time, such as the time associated with the layer of action nodes). In some examples, to determine that two (or more) candidate actions are similar (because they are clustered into the same cluster), an exact match may be required, thresholding may be used (e.g., candidate actions that identify locations that are less than a threshold distance from each other may be identified as similar), and / or k-means clustering may be used.

[0097] If any candidate actions are identified as similar, the process may include associating the predictions from which the two (or more) similar candidate actions were generated / depended on with the same prediction node. In some examples, if multiple predictions are associated with the same prediction node, the process may include determining a weight associated with each different prediction. The weight may indicate the degree to which the prediction is present with that prediction node. Determining the weight may be based at least in part on the similarity of the candidate action generated from the prediction to one or more actions associated with the various associated candidate actions determined from the other predictions. The grouping of predictions into a single prediction node may be used by the process to determine a smaller subset of candidate actions to explore, such as one or the top p candidate actions ranked by cost, where p is a positive integer. For example, the top two candidate actions ranked according to cost may be associated with a prediction node that identifies multiple predictions. Future exploration may be based at least in part on these two candidate actions.

[0098] In yet another example, a prediction of a particular prediction layer may be associated with all of the prediction nodes of that layer. In other words, there may be as many prediction nodes as there are prediction nodes, and each of the prediction nodes may indicate all of the predictions. In such an example, a weight may be associated with each prediction, indicating the likelihood of the prediction occurring and / or how similar the prediction is to the target prediction. Each prediction node may identify a different target prediction, in other words, the prediction node may cycle between various predictions as a basis for comparison.

[0099] Operation 348 may additionally or alternatively include a combining operation that may result in a single predictive node indicative of two or more environmental conditions and / or a single predictive node indicative of a single environmental condition that represents two or more environmental conditions associated with two or more previous predictive nodes. In generating a second predictive node in the same layer as the first predictive node, such as predictive node 352 for predictive node 350, the example process 300 may include determining the similarity of at least two predictive nodes by an ML model. In such an example, the ML model may be a set of perceptrons and / or multiple layers of perceptrons (e.g., CNN), a support vector machine (SVM), or the like. The ML model may be trained to output a score indicative of the similarity between the two environmental conditions. The ML model may be trained according to a supervised method of labeling the environmental conditions as similar or dissimilar. If the score output by the ML model meets or exceeds a threshold, the two (or more) predictive nodes may be combined into a single predictive node; otherwise, the predictive nodes may remain separate. Based on the similarity, further computational resources may be saved because searching both nodes will yield results that are substantially similar.

[0100] In yet another example, the ML model may determine an embedding associated with the prediction node, such as by determining an embedding associated with the environmental state data, which may include transforming the environmental state data into a high-dimensional data structure and projecting the high-dimensional data structure into an embedding space. In such an example, the ML model may include an embedding algorithm, such as Uniform Manifold Approximation and Projection (UMAP), t-Distributed Stochastic Neighbor Embedding (t-SNE), ISO-MAP, Locally Linear Embedding (LLE), or other such algorithms. In an example where the exemplary process 300 determines an embedding for each of two prediction nodes and determines a distance in the embedding space between the two embeddings (e.g., Euclidean distance). If the distance is less than a threshold distance, the prediction nodes may be combined; otherwise, the prediction nodes may remain separate.

[0101] At operation 354, the example process 300 may include determining a second candidate action for controlling the movement of the vehicle based at least in part on the environmental state data indicated by the preceding predictive node, according to any of the techniques discussed herein. For example, determining the second candidate action may be based at least in part on the environmental state data indicated by the predictive node 350. This relationship between the predictive node and the candidate action based thereon is illustrated by an arrow, and in some examples, determining the second candidate action based at least in part on the predictive node 350 may be based at least in part on determining that the simulation that resulted in the updated environmental state data associated with the predictive node 350 did not result in a violation of an operating constraint, that costs were not exceeded, or that there was no impact.

[0102] By way of example and not limitation, the motion constraints may include kinematic and / or qualitative metrics associated with the vehicle's motion, such as a minimum distance between the vehicle and the nearest object, the acceleration required to avoid a collision, the time to collision, deviation from route, whether the jerk and / or acceleration by the vehicle meets or exceeds a threshold jerk and / or threshold acceleration, the predicted likelihood of a collision, etc. The motion constraints may additionally or alternatively include weights associated with various minimum distances that may be used to reduce or increase a reward based at least in part on the minimum distance the simulated autonomous vehicle came from any object during a simulation.

[0103] In some examples, a final cost associated with the first candidate action may be determined after and / or simultaneously with the generation of the prediction node 350. In some examples, the decision to generate a second candidate action may be based at least in part on this final cost. For example, other final costs may be determined in association with the action nodes 342 and / or 344 and / or prediction nodes dependent thereon. The decision to generate a second candidate action branching from the first action node 338 (through the prediction node 35) may be based at least in part on determining that the first action node 338 is associated with a total cost of the action that is less than the total cost of taking another action. The total cost refers to the cost of the candidate action in question and the total cost of any preceding actions in the branch leading to the candidate action in question. In the case of the second candidate action, the total cost would be the final cost associated with the second candidate action plus the final cost associated with the first candidate action.

[0104] The data structure 332 may be updated to include a second action node 356 and action nodes associated with any other candidate actions determined based at least in part on the prediction node 350. Note that Figures 3B and 3C show a simplification of the process in which only one branch of each of the layers is explored. Further actions may be determined from the same prediction node as shown by action nodes 342 and 344 or from separate prediction nodes, further prediction nodes may be determined, and so on. Because the root node 330 itself may contain predicted environmental state data, the prediction nodes 350 and 352 may be considered to be in a second layer of prediction nodes.

[0105] Although the example process 300 may begin operations 358 and / or 360 based at least in part on determining the second candidate action, in some examples operation 358 may additionally or alternatively be determined at least in part based on operation 348.

[0106] In operation 358, the example process 300 may include determining whether the dynamic object may be represented as an inactive or reactive agent according to any of the techniques discussed herein. Operation 358 may include reclassifying at least one of the dynamic objects indicated by a most recent predictive node, such as predictive node 350 or 352. In some examples, operation 358 may be part of operation 348, while in some examples operation 358 may receive a candidate action as an input, which may include waiting for a next candidate action to be generated based at least in part on the predictive node. In the example shown, updating the dynamic object classification for predictive node 350 may include using a second candidate action determined in association with second action node 356. In some examples, an agent filter may be trained to classify dynamic objects without using a candidate action as an input, in which case operation 358 may be part of operation 348.

[0107] Regardless, the track associated with the dynamic object, as indicated by the environmental state data associated with the prediction node, may be provided as an input to the agent filter along with the vehicle's attitude and / or track. Note that at this stage in the tree search, the environmental state data includes the predicted track of the dynamic object. The agent filter may treat the predicted track of the dynamic object as the current track, and the current track identified by the root node will be treated as the past track, or the agent filter may treat the predicted track as the predicted track. In other words, the agent filter may be trained to handle multiple time steps of the track, including the past track, the current track, and / or the predicted track. In some cases, the agent filter may use a portion of the available track associated with a time window, such as the last n seconds relative to the most recent time in the tree search space, where n is a positive number.

[0108] If the classification of a dynamic object has changed, as determined by the agent filter, the classification of that object may be modified in the environmental state data associated with the predictive node. For example, a reactive object may be reclassified as an inert object, or vice versa. In operation 362, the example process 300 may include changing such status in the appropriate predictive node in the data structure. In some examples, this new classification may be used by the agent filter as part of a simulation to determine a subsequent predictive node in operation 364. For example, a not shown predictive node determined based on the second action node 356 may use the updated classification determined in operations 358 and 362 as part of the simulation. If the classification does not change for any of the dynamic objects identified in the environmental state data identified by the predictive node, the example process 300 may continue without modifying that predictive node.

[0109] In operation 360, the exemplary process 300 may include determining a second cost associated with the second candidate action using an upper limit cost, according to any of the techniques discussed herein. As described above, purely by way of example, a lower limit cost is used in operation 346, and alternating between the lower limit cost and the upper limit cost according to a 1:1 ratio would determine that the upper limit cost is used in operation 360. However, as described above, the upper limit cost may be used first, and the ratio may be any other ratio other than 1:1. Regardless, determining the second cost may include using an upper limit cost, and the upper limit cost may be a predetermined cost associated with the default action.

[0110] In operation 366, the example process 300 may include controlling the vehicle based at least in part on a path that includes a first candidate action and a second candidate action based at least in part on a total cost associated therewith, according to any of the techniques discussed herein. For example, the tree search may continue to alternate between determining action nodes and prediction nodes until an end event, such as determining a predefined number of action nodes, a time limit for calculation, a computation or storage limit, a displacement along the route being achieved, a number of time steps being reached (e.g., an action up to the time horizon being determined), determining an action to reach the target location, determining that the path to the target location is below a cost threshold, etc.

[0111] The guidance component may determine one or more successive paths through a set of action nodes when the set of nodes is generated and / or when an end event occurs. In some examples, the guidance component may trace one or more potential paths through a set of action nodes as they are generated and may backtrace the lowest cost path (i.e., from a node at the end of a potential path in the last layer of nodes to a start / root node and / or at the vehicle's current location) to ensure that it is collision-free. Regardless, the guidance system may determine a successive path through a set of nodes that is associated with the lowest cost and / or that advances the most along the route. A successive path of action nodes is a set of nodes that are connected by a dependency relationship in the data structure 332. Intervening predictor nodes are not taken into account for path planning beyond the sub-costs that they may indicate. Two action nodes are dependent when they are connected by an intervening predictor node, which indicates that a lower-level action node (e.g., the second action node 356) starts from the end location of a higher-level action node (e.g., the first action node 338).

[0112] The guidance system may perform a search for a path from a root node to the last layer of the data structure 332. Performing the search may include determining a contiguous set of connections between nodes of various sets of nodes from the root node to an action node in the deepest layer of the data structure 332. Determining the path may include searching for a solution in a multivariate space that maximizes a combination of displacement along the route and lateral / azimuth diversity among the solutions (or satisfies a diversity heuristic) and minimizes a cost based at least in part on a cost map in a given time interval. For example, the search algorithm may include algorithms such as, for example, D*, D*lite, Focused Dynamic A*, A*, LPA*, Dijkstra's algorithm, etc., although other search algorithms for searching and / or generating directed graphs and / or weighted directed graphs may be used. In some examples, the search may be configured with a rule set that may include, for example, one or more rules that specify boundaries within which to determine a route (e.g., the boundaries may be determined based at least in part on sensor data and / or a map), node connection rules (e.g., a node may have only one parent node), etc. In some examples, the search may include determining a directed graph between the nodes of the set of nodes. The directed graph may include connections (e.g., edges) between a first node and a second node and / or weights (e.g., costs) associated with the connections.

[0113] In some examples, before performing a complete search for a route, the guidance component may include determining whether a previous route and / or previous connections determined by the guidance system are feasible (e.g., meets current constraints such as speed, maximum steering angle, and / or bounds, is collision-free, has a cost that is below a cost threshold).

[0114] In some examples, the path determined by the guidance system may be a coarse path. For example, the coarse path may identify the location, direction, speed, and / or curvature of the approach followed by the vehicle at intervals of 1 second or 500 milliseconds, but the vehicle's components may require or be capable of control at finer time intervals (e.g., 10 milliseconds, 100 milliseconds). In other words, the coarse path may not be smooth enough for the vehicle to follow without significant error. In some examples, a first type of processor (e.g., a graphics processing unit (GPU)) may determine the prediction nodes and action nodes and / or determine the path, and a second type of processor may smooth the path generated by the GPU and / or determine a trajectory for controlling the vehicle based at least in part on the smooth path.

[0115] The guidance system discussed herein may identify a route as viable and / or determine a reliability score associated with the route based at least in part on the cost discussed herein. The guidance system may output the route and / or the reliability score, which the autonomous vehicle may use to control the movement of the autonomous vehicle, for example, by generating a trajectory based at least in part on the route. In some examples, the guidance system may output a primary route and / or an incidental route. For example, the guidance system may determine the incidental route based at least in part on generating a set of candidate routes, determining that the set includes two groups of candidate routes (e.g., the two groups may be two distinct homotopic groups) based at least in part on a threshold distance, and selecting a primary route from the first group and an incidental route from the second group. In some examples, the primary route may be selected as the primary route based at least in part on determining that the primary route is associated with a first total cost that is less than a second total cost associated with the incidental route. The primary path may be associated with a first total cost and / or the incidental path may be associated with a second total cost that is less than a cost threshold and / or may be the minimum cost of its respective group associated therewith.

[0116] FIG. 4 shows a three-dimensional representation 400 of four different sets of candidate actions (i.e., trajectories in the illustrated example) generated at four different action layers of the tree search. A first set 402 of candidate actions was generated based at least in part on the vehicle's position 404. These candidate actions 402 may additionally or alternatively be determined based at least in part on the orientation, speed, acceleration, steering speed, environmental state data (discussed in more detail above with respect to static / dynamic objects, etc.) associated with the vehicle's motion, etc., shown in relation to the root node. The space occupied by the vehicle is represented at 406 as a dashed line. FIG. 4 also shows two road edges: road edge 408 and road edge 410. The height of the candidate action indicates the speed and / or acceleration associated with that candidate action.

[0117] The second set of candidate actions 412 may be generated based at least in part on selecting a first candidate action of the first set of candidate actions for exploration 402, and based at least in part on a final position 414, orientation, speed, steering rate, etc. that the first candidate action would have the vehicle achieve upon completing execution of the first candidate action, and environmental state data. The second set of candidate actions 412 may additionally or alternatively be determined based at least in part on environmental state data indicated by a prediction node determined at least in part based on the first candidate action.

[0118] The third set of candidate actions 416 may similarly be based at least in part on the selection of a second candidate action from among the second set of candidate actions 412, the environmental state data generated in association therewith, and / or the final position 418, orientation, speed, steering rate, etc. that the second candidate action will affect. The fourth set of candidate actions 420 may similarly be based at least in part on the selection of a third candidate action from among the third set of candidate actions 416, the environmental state data generated in association therewith, and / or the final position 422, orientation, speed, steering rate, etc. that the third candidate action will affect.

[0119] Representation 400 may be a visual depiction of a determinized sparse partially observable tree (DESPOT) determined according to a partially observable Markov decision process (POMDP).

[0120] 5A and 5B show a top-down view of various candidate actions that may be generated by the tree search for various action nodes at the same level of the tree search. For example, in FIG. 5A, a candidate action 500 may be generated based on environmental state data 502, which may be represented by a predictive node. In FIG. 5B, a candidate action 504 may be generated based on environmental state data 502, which may be represented by the same predictive node. For example, the candidate action 500 may represent a candidate action 340 associated with a first action node 338, and the candidate action 504 may be associated with another action node at the same layer of the data structure, such as action node 344.

[0121] 6A and 6B show a top-down view of two different predicted environmental conditions, environmental condition data 600 and environmental condition data 602, associated with a single predictive node or two different predictive nodes that may be determined based at least in part on the candidate action 500 shown in FIG. 5A. The difference between the environmental conditions may be based at least in part on changing parameters of the simulation of the candidate action 500, as described above with respect to operations 348 and / or 364. In a first simulation that may result in determining the environmental condition data 600, the two reactive objects, vehicle 604 and vehicle 606, may both be assigned a tendency of “nominal” or “conservative,” which may bring vehicle 606 to a complete stop in light of the simulated execution of the candidate action 500 by vehicle 306, and may bring vehicle 604 to a complete stop or slow down, which may result in a final position 608 of vehicle 306.

[0122] Referring to FIG. 6B, environmental state data 602 may be determined at least in part based on a simulation in which two responsive objects, vehicle 604 and vehicle 606, may both be assigned tendencies to be “aggressive” or “inattentional,” which may cause those vehicles to ignore vehicle 306, fail to consider vehicle 306, or attempt to block vehicle 306's path when vehicle 306 performs candidate action 500, resulting in vehicles 604 and 606 being positioned in front of vehicle 306.

[0123] In at least one example, environmental state data 600 may be associated with a first predictive node and environmental state data 602 may be associated with a second predictive node because the two environmental states may be sufficiently different that a combine operation may determine that the two states should be separate. For example, environmental state data 600 may be associated with predictive node 350 and environmental state data 602 may be associated with predictive node 352. Note that both of these predictive nodes are subordinate to the same action node, namely, first action node 338, because both of their associated environmental state data were determined based at least in part on the same action, namely, candidate action 500 (which may correspond to candidate action 340). Note also that these predictive nodes are in the same layer of data structure 332. In additional or alternative examples where the resulting environmental states are more similar, the two environmental states may both be represented by the same predictor node, or one environmental state may represent both environmental states, such as by selecting a representative environmental state or by averaging the environmental states.

[0124] 7A and 7B show a top-down view of two different candidate actions, candidate action 700 and candidate action 702, respectively, that were generated based on the environmental state data 600 shown in FIG. 6A and associated with a first prediction node, e.g., prediction node 350. For example, candidate action 700 may be associated with a second action node 356, and candidate action 702 may be associated with a not-shown action node that, like action node 356, is also subordinate to prediction node 350.

[0125] 7A and 7B also show the result of agent filtering that changes the classification of the dynamic object from a reactive object to an inert object for use by the prediction and / or simulation component in generating the next level prediction node based on the candidate action 700 or the candidate action 702, respectively. Operations 358 and / or 362 may effect this change. Compare the classification of the vehicle 604 in FIGS. 6A and 6B, where the vehicle 604 is classified as a reactive object, to the classification of the vehicle 604 in FIGS. 7A and 7B, where the vehicle 604 is reclassified as an inert object. In such an example, operations 358 and 362 may include classifying the vehicle 604 as an inert object. This may be the case because the vehicle 604 has reached a point where the vehicle 604 will no longer change its behavior based on the actions of the vehicle 306, or is unlikely to change its behavior. This means that in subsequent prediction operations, the vehicle 604 may be simulated more simply, regardless of the candidate actions of the vehicle 306.

[0126] 8A and 8B show further details regarding the determination of candidate actions. FIG. 8A shows a top-down view of a curved road 800, lane criterion 802, and offset criterion 804 used to probabilistically generate actions for a tree search. The road 800 shown may be a single lane for simplicity. The lane criterion 802 may be pre-determined and stored in association with a map, or the lane criterion 802 may be determined by a perception component of the vehicle 306. The offset 806 may be determined probabilistically or iteratively, as discussed in more detail in association with FIG. 8B. The offset may be used to determine an offset criterion 804, which may be a displacement of the lane criterion 802 by an amount defined by the offset 806. The guidance component and / or planning component may determine a candidate action based at least in part on the lane criterion 802 if the offset is zero, which would include generating a trajectory that follows the lane criterion 802. Following lane criteria at or accelerating / decelerating to equal the maximum permitted legal speed (or a predefined speed below the maximum permitted legal speed) may be the default policy used by the tree search, such as in a complex environment determined according to US Pat. No. 6,233,993. This default policy may be used in some examples to determine an upper cost limit. The default policy may be a best case action, such as when all other agents are modeled as being attentive or compliant (yielding to the vehicle), thereby minimizing comfort cost and / or overall cost.

[0127] In examples where the offset is non-zero, determining the candidate actions may include determining a trajectory 808 by drawing a curve, Bezier, gradient, etc. from a starting position 810 to a position on the offset reference 804 that is bounded according to a smoothness constraint associated with a comfort metric and / or steering capability of the vehicle.

[0128] FIG. 8B shows a graph 812 illustrating a sampling technique for determining actions for a tree search that can be used to ensure that a sufficient parameter space is sampled to find a suitable path. This can ensure a diversity of candidate actions indicated by the action nodes, so that a separate action node combining operation is not required. In an additional or alternative example, a combining operation such as that described above with respect to the prediction node can be used to combine similar action nodes to slim down the data structure. The graph 812 shown only shows two parameters that can be changed between candidate actions: velocity 814 in the direction of displacement along the route (represented on the y-axis) and offset 816 from lane reference (represented on the x-axis). It should be understood that additional or alternative parameters such as steering speed, final direction, position in the environment, acceleration, lateral and / or longitudinal speed (instead of or in addition to the speed along the route), etc. can be used to generate candidate actions. The graph 812 includes samples such as sample 818 shown as diamonds, which are sampled according to 3 meters per second and 0.2 meters intervals. Any other interval may be used to sample the space, and the interval may be a predetermined value based on the type of parameter being sampled and may be selected such that the candidate actions are sufficiently different. In some examples, the sample may be limited to an area, such as area 820. Area 820 may be determined based at least in part on sensory data, environmental state data, the current trajectory, and / or previous actions. In the example shown, the vehicle may currently be moving fast, and therefore it may not be feasible to slow down to 0 meters per second in the next time step. At a maximum, the vehicle may be able to slow down to 8 meters per second in the next time step. Also, the higher the vehicle's speed, the more likely the vehicle may be to violate the curvature / steering speed constraint, which may be speed dependent, thereby limiting the offset value.In other words, the faster the vehicle is moving, the narrower the candidate actions may be (laterally) to not exceed lateral acceleration or jerk.

[0129] Example clauses A: receiving route data associated with a start location and an end location in an environment; receiving sensor data from a sensor; determining a first candidate action for controlling movement of the vehicle based at least in part on the sensor data and the route data; determining a first cost associated with the first candidate action, the first cost based at least in part on a lower bound cost, the lower bound cost being an estimate of a minimum cost of an action to change a first state of the vehicle at a time associated with a beginning of the first candidate action; determining a first prediction associated with a first state of the environment based at least in part on the first candidate action and the sensor data, the first prediction including a first state of an object in the environment at a future time; determining a second cost associated with the second candidate action, the second cost based at least in part on a ceiling cost, the ceiling cost being an estimate of a cost for performing a default action; alternatingly applying one of the floor cost or the ceiling cost to one or more subsequent candidate actions; and controlling the vehicle based at least in part on determining that a first total cost including the first candidate action and the second candidate action and that the first total cost including the first cost and the second cost is less than a threshold or less than a second total cost associated with a second path including at least one candidate action different from at least one of the first candidate action or the second candidate action.

[0130] B: The method of paragraph A, further comprising the steps of determining, by a machine learning model and based at least in part on the sensor data, that the object is a reactive object, and determining a first prediction based at least in part on running a first simulation using the determination that the object is a reactive object, wherein running the first simulation using the determination that the object is a reactive object includes determining a movement of a representation of the object based at least in part on the first candidate action.

[0131] C: The method of any of paragraphs A or B, further comprising determining, by a machine learning model and based at least in part on the sensor data, that the object is an inactive object, and determining a first prediction based at least in part on modeling a movement of the inactive object, wherein modeling the movement of the inactive object includes determining a movement of the inactive object based at least in part on a state of the object and to the exclusion of a first candidate action.

[0132] D: The method of any one of paragraphs A-C, further comprising the steps of allocating the first candidate action and the second candidate action to the same prediction node of the data structure, the first prediction and the second prediction being associated with the same prediction layer as the first prediction based at least in part on: determining a first set of candidate actions based at least in part on the first prediction and a second set of candidate actions based at least in part on the second prediction; determining a first action from among the first set associated with a first minimum cost from among the costs associated with the first set; determining a second action from among the second set associated with a second minimum cost from among the costs associated with the first set; and determining that the first action and the second action are associated with positions that are within a distance from each other that is less than a threshold distance.

[0133] E: The method of any one of paragraphs A-D, further comprising one of determining a likelihood that the object will modify its behavior in response to one or more of the first candidate action or the second candidate action based at least in part on the sensor data, and deciding to classify the object as a reactive agent based at least in part on the likelihood meeting or exceeding a threshold, or deciding to classify the object as an inactive agent based at least in part on the likelihood being equal to or less than a threshold.

[0134] F: The method of any one of paragraphs A-E, wherein the lower bound cost is zero and the upper bound cost is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

[0135] G: A system comprising one or more processors and a memory storing processor-executable instructions, the processor-executable instructions, when executed by the one or more processors, causing the system to: receive route data associated with a start location and an end location in an environment; receive sensor data from a sensor; determine a first candidate action for controlling movement of a vehicle based at least in part on the sensor data and the route data; determine a first cost associated with the first candidate action, the first cost being based at least in part on a lower bound cost, the lower bound cost being an estimate of a minimum cost of an action to change a first state of the environment at a time associated with a beginning of the first candidate action; and determine a first cost associated with the first candidate action, the first cost being based at least in part on a lower bound cost, the lower bound cost being an estimate of a minimum cost of an action to change a first state of the environment at a time associated with a beginning of the first candidate action and the sensor data. determining a first prediction associated with a default action, the first prediction including a first state of an object in an environment at a future time; determining a second candidate action for controlling movement of the vehicle based at least in part on the first prediction and the route data; determining a second cost associated with the second candidate action, the second cost based at least in part on a ceiling cost, the ceiling cost being an estimate of the cost of performing a default action; alternatingly applying one of the floor cost or the ceiling cost to one or more subsequent candidate actions; and controlling the vehicle based at least in part on determining a path including the first candidate action and the second candidate action and a first total cost associated with the first candidate action and the second candidate action.

[0136] H: The system described in paragraph G, wherein the operations further include determining, by the machine learning model and based at least in part on the sensor data, that the object is a reactive object, and determining a first prediction based at least in part on running a first simulation using the determination that the object is a reactive object, wherein running the first simulation using the determination that the object is a reactive object includes determining a movement of a representation of the object based at least in part on the first candidate action.

[0137] I: The system of any of paragraphs G or H, wherein the operations further include determining that the object is an inactive object by a machine learning model and based at least in part on the sensor data, and determining a first prediction based at least in part on modeling a movement of the inactive object, wherein modeling the movement of the inactive object includes determining a movement of the inactive object based at least in part on a state of the object and to the exclusion of a first candidate action.

[0138] J: The system of any one of paragraphs G-I, wherein the operation further includes allocating the first candidate action and the second candidate action to a same prediction node of the data structure, the first prediction and the second prediction being associated with a same prediction layer as the first prediction based at least in part on: determining a first set of candidate actions based at least in part on the first prediction and a second set of candidate actions based at least in part on the second prediction; determining a first action from among the first set associated with a first minimum cost from among the costs associated with the first set; determining a second action from among the second set associated with a second minimum cost from among the costs associated with the first set; and determining that the first action and the second action are associated with locations that are within a distance from each other that is less than a threshold distance.

[0139] K: The system of any one of paragraphs G-J, wherein the operations further include determining, based at least in part on the sensor data, a likelihood that the object will modify its behavior in response to one or more of the first candidate action or the second candidate action, and determining to classify the object as a reactive agent based at least in part on the likelihood meeting or exceeding a threshold, or determining to classify the object as an inactive agent based at least in part on the likelihood being equal to or less than a threshold.

[0140] L: The system of any one of paragraphs G-K, wherein the lower bound cost is zero and the upper bound cost is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

[0141] M: The system of any one of paragraphs G-L, wherein controlling the vehicle based at least in part on the path is further based at least in part on determining that the first total cost is less than a threshold or less than a second total cost associated with a second path that includes at least one candidate action that is different from at least one of the first candidate action or the second candidate action.

[0142] N: a non-transitory computer-readable medium storing processor-executable instructions, the processor-executable instructions, when executed by one or more processors, to cause the one or more processors to: receive route data associated with a start location and an end location in an environment; receive sensor data from a sensor; determine a first candidate action for controlling movement of a vehicle based at least in part on the sensor data and the route data; determine a first cost associated with the first candidate action, the first cost being based at least in part on a floor cost or an upper cost; and determine a first prediction associated with a first state of the environment based at least in part on the first candidate action and the sensor data; a first prediction including a first state of an object in an environment at a future time; determining a second candidate action for controlling movement of the vehicle based at least in part on the first prediction and the route data; and determining a second cost associated with the second candidate action, the second cost being based at least in part on a floor cost or an upper cost; alternatingly applying one of the floor cost or the upper cost to one or more subsequent candidate actions; and controlling the vehicle based at least in part on determining a path including the first candidate action and the second candidate action and a first total cost associated with the first candidate action and the second candidate action.

[0143] O: The non-transitory computer-readable medium of paragraph N, wherein the operations further include determining, by the machine learning model and based at least in part on the sensor data, that the object is a reactive object, and determining a first prediction based at least in part on running a first simulation using the determination that the object is a reactive object, wherein running the first simulation using the determination that the object is a reactive object includes determining a movement of a representation of the object based at least in part on the first candidate action.

[0144] P: The non-transitory computer-readable medium of any of paragraphs N or O, wherein the operations further include determining that the object is an inactive object by a machine learning model and based at least in part on the sensor data, and determining a first prediction based at least in part on modeling a movement of the inactive object, wherein modeling the movement of the inactive object includes determining a movement of the inactive object based at least in part on a state of the object and to the exclusion of a first candidate action.

[0145] Q: The non-transitory computer-readable medium of any one of paragraphs N-P, wherein the operation further includes allocating a first candidate action and a second candidate action to a same prediction node of the data structure, the first prediction and the second prediction being associated with a same prediction layer as the first prediction based at least in part on: determining a first set of candidate actions based at least in part on the first prediction and a second set of candidate actions based at least in part on the second prediction; determining a first action from among the first set associated with a first minimum cost from among the costs associated with the first set; determining a second action from among the second set associated with a second minimum cost from among the costs associated with the first set; and determining that the first action and the second action are associated with locations that are within a distance from each other that is less than a threshold distance.

[0146] R: The non-transitory computer-readable medium of any one of paragraphs N-Q, wherein the operations further include determining, based at least in part on the sensor data, a likelihood that the object will modify its behavior in response to one or more of the first candidate action or the second candidate action, and determining to classify the object as a reactive agent based at least in part on the likelihood meeting or exceeding a threshold, or determining to classify the object as an inactive agent based at least in part on the likelihood being equal to or less than a threshold.

[0147] S: The non-transitory computer-readable medium of any one of paragraphs N-R, wherein the lower bound cost is an estimate of a minimum cost of an action to change the first state of the vehicle at a time associated with the beginning of the first candidate action, and the upper bound cost is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

[0148] T: A non-transitory computer-readable medium as described in any one of paragraphs N-S, wherein controlling the vehicle based at least in part on the path is further based at least in part on determining that the first total cost is less than a threshold or less than a second total cost associated with a second path that includes at least one candidate action that is different from at least one of the first candidate action or the second candidate action.

[0149] Although the example provisions set forth above are described with respect to one particular implementation, it should be understood in the context of this document that the contents of the example provisions may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. In addition, any of the examples A-T may be implemented alone or in combination with any other one or more of the examples A-T.

[0150] Conclusion Although the present subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the present subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0151] The components described herein correspond to instructions that can be stored in any type of computer-readable medium and can be implemented in software and / or hardware. All of the methods and processes described above can be embodied in and fully automated through software code components and / or computer executable instructions executed by one or more computers or processors, hardware, or any combination thereof. Some or all of the methods can alternatively be embodied in dedicated computer hardware.

[0152] At least some of the processes discussed herein are illustrated as logical flow graphs, each of whose operations represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or autonomous vehicle to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0153] Conditional language such as "may," "could," "may," or "might," among others, is understood in context to indicate that a particular example includes a particular feature, element, and / or step while other examples do not, unless specifically stated otherwise. Thus, such conditional language is not generally intended to imply that a particular feature, element, and / or step is somehow required for one or more examples, or that one or more examples necessarily include logic for determining, with or without user input or prompting, whether a particular feature, element, and / or step is included in or will be performed in any particular example.

[0154] Transitive language such as the phrase "at least one of X, Y, or Z" should be understood to indicate that the item, term, etc. can be either X, Y, or Z, or any combination thereof, including a plurality of each of the elements, unless specifically stated otherwise. "a" means singular and plural, unless expressly described as singular.

[0155] Any routine description, element, or block in the flow diagrams described herein and / or shown in the accompanying figures should be understood as potentially representing a module, segment, or portion of code that includes one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions may be omitted or performed out of the order shown or discussed, substantially synchronously, in reverse order, with additional operations, or with omitted operations, etc., depending on the functionality involved, as would be understood by one of ordinary skill in the art. It should be noted that the term substantially may indicate a range. For example, substantially simultaneously may indicate that two activities occur within a time range of each other, substantially the same dimension may indicate that two elements have dimensions within each other, and so on.

[0156] Many variations and modifications may be made to the above-described examples, and it is understood that the elements thereof are among the other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. one or more processors; and a memory storing processor-executable instructions that, when executed by the one or more processors, provide the system with: receiving route data associated with a start location and an end location in the environment; receiving sensor data from a sensor; determining a first candidate action for controlling movement of a vehicle based at least in part on the sensor data and the route data; determining a first cost associated with the first candidate action, the first cost being based at least in part on a lower bound cost, the lower bound cost being an estimate of a minimum cost of an action to change a first state of the vehicle at a time associated with a beginning of the first candidate action; determining a first prediction associated with a first state of the environment based at least in part on the first candidate action and the sensor data, the first prediction including a first state of an object in the environment at a future time; determining second candidate actions for controlling movement of the vehicle based at least in part on the first prediction and the route data; determining a second cost associated with the second candidate action, the second cost being based at least in part on an upper limit cost, the upper limit cost being an estimate of the cost of performing a default action; Alternately applying one of the lower bound cost or the upper bound cost to one or more subsequent candidate actions; a path including the first candidate action and the second candidate action; and, determining a first total cost associated with the first candidate action and the second candidate action; and controlling the vehicle based at least in part on the

2. The operation determining, by a machine learning model and based at least in part on sensor data, that the object is a reactive object; determining the first prediction based at least in part on running a first simulation using the determination that the object is a reactive object; 2. The system of claim 1, wherein performing the first simulation using the determination that the object is a reactive object includes determining a movement of a representation of the object based at least in part on the first candidate action.

3. The operation determining, by a machine learning model and based at least in part on the sensor data, that the object is an inert object; determining the first prediction based at least in part on modeling the movement of the inert object; 3. The system of claim 2, wherein modeling the movement of the inert object includes determining the movement of the inert object based at least in part on a state of the object and to the exclusion of the first candidate action.

4. The operation is to allocate the first candidate action and the second candidate action to the same prediction node of a data structure, wherein the first prediction and the second prediction are: determining a first set of candidate actions based at least in part on the first prediction and a second set of candidate actions based at least in part on the second prediction; determining a first action from the first set associated with a first smallest cost from among the costs associated with the first set; determining a second action from the second set associated with a second smallest cost from among the costs associated with the first set; determining that the first action and the second action are associated with locations that are within a distance from each other that is less than a threshold distance; The system of claim 3 , further comprising allocating a first prediction associated with a same prediction layer as the first prediction based at least in part on

5. The operation determining, based at least in part on the sensor data, a likelihood that the object will modify a behavior in response to one or more of the first candidate action or the second candidate action; determining to classify the object as a reactive agent based at least in part on the likelihood meeting or exceeding a threshold; or and determining to classify the object as an inactive agent based at least in part on the likelihood being less than or equal to the threshold.

6. 6. The system of claim 5, wherein the lower bound cost is zero and the upper bound cost is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

7. 7. The system of claim 6, wherein controlling the vehicle based at least in part on the route is further based at least in part on determining that the first total cost is less than a threshold or less than a second total cost associated with a second route that includes at least one candidate action that is different from at least one of the first candidate action or the second candidate action.

8. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to: receiving route data associated with a start location and an end location in the environment; receiving sensor data from a sensor; determining a first candidate action for controlling movement of a vehicle based at least in part on the sensor data and the route data; determining a first cost associated with the first candidate action, the first cost being based at least in part on a floor cost or an upper cost limit; determining a first prediction associated with a first state of the environment based at least in part on the first candidate action and the sensor data, the first prediction including a first state of an object in the environment at a future time; determining second candidate actions for controlling movement of the vehicle based at least in part on the first prediction and the route data; determining a second cost associated with the second candidate action, the second cost being based at least in part on the floor cost or the ceiling cost; Alternately applying one of the lower bound cost or the upper bound cost to one or more subsequent candidate actions; a path including the first candidate action and the second candidate action; and based at least in part on determining a first total cost associated with the first candidate action and the second candidate action; and controlling the vehicle.

9. The operation determining, by a machine learning model and based at least in part on the sensor data, that the object is a reactive object; determining the first prediction based at least in part on running a first simulation using the determination that the object is a reactive object; 10. The non-transitory computer-readable medium of claim 8, wherein performing the first simulation using the determination that the object is a reactive object includes determining a movement of a representation of the object based at least in part on the first candidate action.

10. The operation determining, by a machine learning model and based at least in part on the sensor data, that the object is an inert object; determining the first prediction based at least in part on modeling the movement of the inert object; 10. The non-transitory computer-readable medium of claim 9, wherein modeling the movement of the inert object comprises determining the movement of the inert object based at least in part on a state of the object and to the exclusion of the first candidate action.

11. The operation is to allocate the first candidate action and the second candidate action to the same prediction node of a data structure, wherein the first prediction and the second prediction are: determining a first set of candidate actions based at least in part on the first prediction and a second set of candidate actions based at least in part on the second prediction; determining a first action from the first set associated with a first smallest cost from among the costs associated with the first set; determining a second action from the second set associated with a second smallest cost from among the costs associated with the first set; determining that the first action and the second action are associated with locations that are within a distance from each other that is less than a threshold distance; 11. The non-transitory computer-readable medium of claim 10, further comprising allocating a first prediction associated with a same prediction layer as the first prediction based at least in part on

12. The operation determining, based at least in part on the sensor data, a likelihood that the object will modify a behavior in response to one or more of the first candidate action or the second candidate action; determining to classify the object as a reactive agent based at least in part on the likelihood meeting or exceeding a threshold; or and determining to classify the object as an inactive agent based at least in part on the likelihood being less than or equal to the threshold.

13. the lower bound cost is an estimate of a minimum cost of an action to change a first state of the vehicle at a time associated with a beginning of the first candidate action; 13. The non-transitory computer-readable medium of claim 12, wherein the upper cost limit is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

14. 14. The non-transitory computer-readable medium of claim 13, wherein controlling the vehicle based at least in part on the route is further based at least in part on determining that the first total cost is less than a threshold or less than a second total cost associated with a second route that includes at least one candidate action that is different from at least one of the first candidate action or the second candidate action.

15. 15. The non-transitory computer-readable medium of claim 14, wherein the lower bound cost is zero and the upper bound cost is determined based at least in part on at least one of a cost associated with the second candidate action or a default trajectory.

Images