Active prediction based on object trajectories

Abstract

Techniques are described herein for accurately predicting and avoiding collisions with objects detected within a vehicle's environment. The vehicle computing device can implement a model to output data indicative of the cost of a potential intersection between the object and the vehicle in the future. The model can use a control policy and a time-step integrator to determine whether the object may intersect with the vehicle, and if so, the techniques can include predicting a vehicle action by the vehicle computing device to control the vehicle.

Description

[Technical Field]

[0001] The present invention relates to techniques for active prediction based on object trajectories. [Background technology]

[0002] This application claims priority to U.S. Patent Application No. 17 / 351,642, entitled "Active Prediction Based on Object Traffic," filed June 18, 2021, which is incorporated herein by reference in its entirety.

[0003] Planning systems in autonomous and semi-autonomous vehicles determine actions the vehicle should take in its operating environment. The vehicle's actions may be determined in part based on avoiding objects present in the environment. For example, actions may be generated by the planning system to yield to pedestrians or change lanes to avoid another vehicle on the road. Perception systems utilize sensor data from sensors to "see" the environment, allowing the planning system to determine the impact of detected objects on the vehicle's potential actions. However, the complexity of such systems can hinder testing their reliability, especially when applied to increasingly complex scenarios. [Brief explanation of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features. [Figure 1] FIG. 1 illustrates an example environment in which an example vehicle applies a model to predict intersection values ​​indicative of the likelihood of a collision with one or more example objects. [Figure 2]FIG. 1 illustrates another example environment in which one or more models determine potential interactions between one or more objects and a vehicle at a future time. [Figure 3] FIG. 1 illustrates another example environment in which one or more models determine potential vehicle and / or object states at future times. [Figure 4] FIG. 1 is a block diagram of an example system for implementing the techniques described herein. [Figure 5A] 1 is a first portion of a flowchart illustrating an example process for determining potential intersection points using one or more example models. [Figure 5B] 10 is a second portion of a flowchart illustrating an example process for determining potential intersections using one or more example models. DETAILED DESCRIPTION OF THE INVENTION

[0005] Described herein are techniques for predicting the trajectories of objects and / or vehicles based on potential interactions with other objects. The techniques include a vehicle computing device implementing a model to determine realistic interactions between multiple objects and improve processing when determining actions for an autonomous vehicle. For example, the model may receive a set of predicted object trajectories associated with the objects, receive and / or determine a vehicle trajectory, determine a predicted intersection between the object trajectories and / or vehicle trajectories based on the predicted trajectories associated with the objects, determine potential actions for the autonomous vehicle, and avoid the objects based on a control policy. In various examples, the model may also determine the cost of each vehicle action for controlling the autonomous vehicle. Implementing the techniques described herein allows for accurate and efficient processing and identification of future states of objects and vehicles, thereby improving overall vehicle safety.

[0006] In general, a model (e.g., an active prediction component) implemented by a vehicle computing device may provide functionality for determining interactions between a vehicle (or object) and another object in a future environment. For example, the model may determine an object that is most likely to intersect with the vehicle trajectory in the future based on identifying an intersection between the object trajectory and the vehicle trajectory. Furthermore, the model may predict a future state of the vehicle relative to the object trajectory based at least in part on a control policy (e.g., information identifying rules of the road between the object and / or vehicle). The future state may represent actions to be performed by the vehicle at different times in the future. In some examples, the vehicle may be controlled within the environment based at least in part on data output by the model.

[0007] In some examples, the prediction component can determine predicted trajectories associated with objects in the environment. For example, in an environment having a vehicle (such as an autonomous vehicle), a first object, and a second object, the prediction component can determine a predicted trajectory associated with each object. Additionally, the planning system can determine several potential actions associated with the vehicle. In some cases, the predicted trajectory may be based on map data and possible actions associated with the object. However, if the object follows the first predicted trajectory (or the vehicle takes the first action), it may affect the behavior of other objects in the environment. Thus, the technique includes determining some or all pairwise combinations of trajectories of vehicle / object pairs or object / object pairs and determining potential interactions (e.g., intersections) associated with such pairs (or a selected subset thereof). The environment can be simulated, and the predicted trajectories of the objects and / or possible actions can be updated or adjusted to take into account the potential interactions of the objects and / or vehicles, which can provide more accurate predictions and simulations of objects in the environment.

[0008] In some examples, the model may determine which of multiple objects will arrive at an intersection with the vehicle first based on object state data associated with each object and vehicle state data associated with the vehicle. For example, the model may receive object state data (e.g., position data, orientation data, path data, velocity data, speed data, acceleration data, yaw rate data, or rotation rate data associated with an object) and / or vehicle state data (e.g., position data, orientation data, path data, velocity data, speed data, acceleration data, yaw rate data, or rotation rate data associated with the vehicle) from a perception component of the vehicle that uses one or more sensors to detect and predict objects surrounding the vehicle's environment. In various examples, the model may receive sensor data, map data, or other input data and identify interactions between two or more objects and / or between an object and the vehicle. In the example of a predicted intersection between two objects, the model may identify the object that will arrive at the intersection with the vehicle first based on dynamics, kinematics, and / or a grid, and predict vehicle actions (e.g., steering actions, acceleration actions, etc.) to avoid the object that is most likely to interact with the vehicle.

[0009] In some examples, the techniques described herein include implementing a model to determine intersection information usable by a vehicle computing device and control the vehicle (e.g., control steering, braking, acceleration, etc.). For example, the vehicle computing device can receive intersection information from the model for use in planning operations, such as determining candidate trajectories for the vehicle. In various examples, the vehicle can receive indications of potential intersections with objects usable by the vehicle and quickly and / or preemptively avoid the objects or update the predicted trajectories of the objects, thereby improving vehicle safety. Additionally, aspects of the processing operations may be parallelized and input to parallel processor units (e.g., GPUs) for efficient processing. Thus, implementing the techniques described herein can efficiently utilize available computational resources (e.g., memory and / or processor allocation or usage) while also improving prediction accuracy.

[0010] In various examples, the model may determine a predicted intersection between the object and the vehicle based at least in part on an occupancy grid including a plurality of grid points representing discretized portions (which may otherwise be referred to herein as pixels) in the environment. In various examples, the model may process object state data, vehicle state data, sensor data, map data, or other input data to determine potential actions by the vehicle to avoid intersecting with the object (e.g., future object states and future vehicle states may be determined in parallel by one or more GPUs). Additional details for determining potential intersections are described herein, including in connection with FIGS. 2 and 3.

[0011] In some examples, the vehicle may include an autonomous or semi-autonomous vehicle having a vehicle computing device configured to receive sensor data from one or more sensors of the vehicle. The vehicle may detect objects using the one or more sensors while navigating within the environment. The objects may include static objects (e.g., ground level, buildings, bridges, signs, etc.) and dynamic objects such as other vehicles (e.g., cars, trucks, motorcycles, mopeds, etc.), pedestrians, bicyclists, etc. In some examples, the objects may be detected based on sensor data from sensors of the vehicle (e.g., cameras, motion detectors, lidar sensors, radar sensors, etc.). As yet another example, the objects may be detected based on sensor data received from remote sensors, such as sensors associated with another vehicle or sensors located within the environment configured to share data with multiple vehicles. The sensor data representing the detected objects may be utilized to determine input data usable by the model and to predict intersection probabilities associated with the objects.

[0012] In some examples, the number of pairwise combinations of object trajectories and / or object actions can be large. For example, if three actions are considered for the vehicle and three trajectories are considered for the object, this can include nine possible combinations. In some examples, potential interactions for such combinations can be determined by implementing the processing on a parallel processor, such as a GPU, and / or by multiple GPUs in parallel, thereby more efficiently processing potential interactions in real time.

[0013] In some examples, the model may define processing resources (e.g., processor amount, processor cycles, processor cores, processor location, processor type, etc.) to use to predict intersection probabilities associated with potential intersections with the vehicle. The vehicle computing device implementing the model may have different processors (e.g., central processing units (CPUs), graphics processing units (GPUs), multi-core processors, etc.). The model may define processing resources and utilize the most efficient (e.g., using the least amount of computation time) processor to output predictions. In some examples, the model may predict intersection values ​​by processing data associated with objects and / or vehicles using a GPU, a CPU, or a combination thereof. In this manner, the model may be defined to utilize processing resources that enable the model to perform predictions in the least amount of time (e.g., use intersection values ​​in vehicle planning considerations). Thus, the model may maximize available processing resources and enable more predictions that may improve how the vehicle navigates in relation to objects.

[0014] As described herein, a model may represent a machine learning model, a statistical model, or a combination thereof. That is, a model may refer to a machine learning model that learns from a training dataset to improve the accuracy of an output (e.g., a prediction). Additionally or alternatively, a model may refer to a statistical model that represents a logical and / or mathematical function that generates an approximation that can be used to make a prediction.

[0015] The techniques described herein can improve the functionality of a vehicle's computing device in many ways. For example, a model may identify which objects to handle in the environment, prevent unnecessary actions by the vehicle, and / or improve predictions related to the vehicle's behavior. In some examples, a model improves vehicle functionality and safety by preventing the vehicle from taking additional actions that are not required based on the state of the environment surrounding the vehicle. Furthermore, the techniques described herein can improve passenger comfort and / or vehicle safety, such as avoiding sudden braking or steering when not required. The techniques can include a model that optimizes available computing resources by performing actions that limit the impact on available resources (compared to not implementing the model). For example, utilizing output data from a model by a vehicle computing device can improve the accuracy and / or reduce latency for the vehicle to respond to potential collisions in the environment.

[0016] In various examples, implementing a model can improve vehicle safety by outputting an indication of multiple predicted trajectories associated with one or more objects in the future. For example, a determined path or trajectory (e.g., a candidate trajectory) for a vehicle may be based at least in part on consideration of multiple predicted trajectories for each object. In some examples, multiple trajectories for multiple objects can be processed by a model to identify which objects and associated trajectories are most likely to interact with the vehicle, thereby allowing a planning component to select a path or trajectory for the vehicle that is most likely to avoid the objects. By implementing a model, consideration of multiple object trajectories and potential actions by each object and / or by the vehicle in a simulation can be determined substantially in real time before the safest possible route for the vehicle to follow is determined.

[0017] The techniques described herein can be implemented in several ways. Examples are presented below with reference to the figures. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems and are not limited to autonomous vehicles. In another example, the techniques can be utilized in an aviation or nautical context, or in any system that uses sensor data. Furthermore, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0018] 1 illustrates an autonomous vehicle (vehicle 102) in an environment 100 in which an example predictive model (active prediction component 104) may receive input data 106 and determine output data 108. A vehicle computing device (e.g., vehicle computing device 404) and / or a remote computing device (e.g., computing device 436) may implement the active prediction component 104 of the vehicle 102. Although described as separate systems, in some examples, the prediction techniques described herein may be implemented by other vehicle systems, components, and / or computing devices. For example, as described in further detail with respect to FIG. 4, the cross-prediction techniques described herein may be implemented at least in part by or in association with the positioning component 420, the perception component 422, the prediction component 424, and / or the planning component 426.

[0019] In some examples, the input data 106 may include one or more of vehicle state data, object state data, sensor data, map data, predicted object trajectory, and / or predicted vehicle trajectory. For example, the active prediction component 104 may receive the input data 106 (e.g., predicted object trajectory, predicted vehicle trajectory, object position, vehicle position, etc.) and generate output data 108 representing costs of one or more intersections between an object and a vehicle in the environment 100. In such examples, the active prediction component 104 may receive input data representing object state data (e.g., position data, orientation data, path data, velocity data, speed data, acceleration data, yaw rate data, or rotation rate data associated with the object) and / or vehicle state data (e.g., position data, orientation data, path data, velocity data, speed data, acceleration data, yaw rate data, or rotation rate data associated with the vehicle). In various examples, the active prediction component 104 may receive map data representing static features of the environment 100 (which, in at least some examples, may also include designations for lane markings, lane speeds, road control information (e.g., stop signs, traffic signals, crosswalks, school zones, speed bumps, etc.)).

[0020] In various examples, the active prediction component 104 may determine output data 108 while the vehicle 102 is navigating within the environment 100. For example, the output data 108 may represent future states of one or more objects or the vehicle 102 for different times in the future as the vehicle 102 navigates to a destination within the environment 100. In various examples, the active prediction component 104 may receive input data 106 from a vehicle computing device and / or one or more components of a computing device remote from the vehicle 102 and may determine an action associated with one of a plurality of potential trajectories associated with the input data 106.

[0021] In various examples, a vehicle computing device associated with the active prediction component 104 may be configured to receive sensor data describing objects in the environment 100, such as via a perception component (e.g., perception component 422). In some examples, the vehicle computing device may detect, infer, estimate, or otherwise determine object state data describing characteristics of objects in the environment 100. For example, such object state data may include position, velocity, acceleration, size, semantic type, etc. In some examples, the sensors may include sensors mounted on the vehicle 102 and may include, but are not limited to, ultrasonic sensors, radar sensors, light detection and ranging (LIDAR) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, etc. In some examples, the sensors may include one or more remote sensors, such as, for example, sensors mounted on another autonomous vehicle and / or sensors mounted on the environment 100. In various examples, the vehicle 102 may be configured to transmit and / or receive data from other autonomous vehicles. The data may include sensor data and / or state data, such as sensor data associated with the environment 100. Additional examples of determining object state data and vehicle state data based on sensor data can be found in U.S. patent application Ser. No. 16 / 151,607, filed Oct. 4, 2018, entitled "Trajectory Prediction on Top-Down Scenes," which is incorporated herein by reference in its entirety for all purposes. Additional examples of tracking objects can be found in U.S. patent application Ser. No. 16 / 147,328, filed Sep. 28, 2018, entitled "Image Embedding for Object Matching," which is incorporated herein by reference in its entirety for all purposes.Additional examples of selecting bounding boxes can be found in U.S. Patent Application No. 16 / 201,842, filed November 27, 2018, entitled "Bounding Box Selection," which is incorporated herein by reference in its entirety for all purposes.

[0022] As mentioned above, in various examples, the active prediction component 104 may receive map data representing fixed features of the environment, such as one or more of roads, intersections, crosswalks, traffic signals, centerlines, road boundaries, speed bumps, school zones, etc. The map data may be received from one or more components of a vehicle computing device or a remote computing device (e.g., a computing device associated with an autonomous vehicle in a fleet of vehicles or another computing device remote from the vehicle 102).

[0023] In some examples, the vehicle computing device may be configured to detect objects in the environment 100, such as object 110 and object 112 (e.g., vehicles), and object 114 (e.g., pedestrian), to name a few. In some examples, the vehicle computing device may be configured to receive, detect, infer, estimate, or otherwise determine one or more trajectories (e.g., direction, speed, acceleration, etc.) for each detected object. As shown in FIG. 1 , object 110 is associated with object trajectories 116(1) and 116(2), object 112 is associated with object trajectories 118(1) and 118(2), object 114 is associated with object trajectory 120, and vehicle 102 is associated with vehicle trajectories 122(1) and 122(2) determined by the vehicle computing device (e.g., using perception component 422, prediction component 424, or another model). In some examples, the active prediction component 104 may receive path information associated with the aforementioned object trajectories (e.g., 116(1), 116(2), 118(1), 118(2), 120, 122(1), and 122(2)) from a machine learning model. While FIG. 1 shows one or two trajectories associated with various objects, any number of objects may be detected and any number of object trajectories may be predicted for each object. Additional details using top-down representations are described in U.S. Patent Application No. 16 / 151,607, filed October 4, 2018, entitled "Trajectory Prediction on Top-Down Scenes," which is incorporated herein by reference in its entirety for all purposes.

[0024] In some examples, the active prediction component 104 can determine potential intersections for one or more paths associated with the vehicle 112 (e.g., object trajectories 118(1) and 118(2) or other object trajectories). In such examples, the intersections output by the active prediction component 104 may indicate potential intersections between the vehicle 112 and the vehicle 102 at the intersections. Further description of determining potential intersections is described in connection with FIG. 2 and throughout this disclosure. The trajectories can be updated based on the potential intersections, and a cost for the vehicle can be determined. For example, the object trajectories most likely to interact with the vehicle 102 can be selected for further processing, such as running a simulation. Further description of the active prediction component 104 selecting a trajectory and / or the active prediction component 104 determining a cost is described in FIG. 3 and elsewhere.

[0025] In some examples, the active prediction component 104 may determine vehicle states and / or object states at different times in the future. The active prediction component 104 may determine actions of the vehicle 110, the vehicle 112, the pedestrian 114, and / or the vehicle 102 during the simulation. For example, actions (e.g., steering actions, braking actions, acceleration actions, etc.) may be determined for a period of time during the simulation (e.g., every second during a 4-second simulation). In various examples, the active prediction component 104 may select, determine, or otherwise predict a vehicle trajectory that the vehicle 102 can use to navigate one or more paths in the environment 100 (e.g., object trajectories 118(1) and 118(2) associated with the vehicle 112). In various examples, the output data 108 may indicate a likelihood that the vehicle 112 will intersect with at least a portion of the vehicle 102. Additional discussion regarding future vehicle states and future object states may be found in FIG. 3 and elsewhere.

[0026] In some examples, the vehicle computing device may control the vehicle 102 in an environment (e.g., determine candidate trajectories to avoid objects in the environment) based at least in part on the output data 108. For example, the active prediction component 104 may provide functionality to identify objects that are most likely to cause an intersection and communicate intersection information about the identified objects to other components of the vehicle computing device. Thus, potential intersections (e.g., the probability or likelihood of an intersection between an object and the vehicle) may be considered in operations (e.g., simulation, estimated state, interaction modeling, collision estimation, etc.) or other vehicle control planning operations (e.g., determined by a planning component) to quickly and / or preemptively avoid the object, thereby improving vehicle safety. Additional details for determining and utilizing the output data 108 are described throughout this disclosure. Additional details for performing simulations and / or generating trajectories are described in U.S. patent application Ser. No. 15 / 632,147, filed June 23, 2017, entitled "Trajectory Generation Using Temporal Logic and Tree Search," and U.S. patent application Ser. No. 15 / 843,512, filed December 15, 2017, entitled "Trajectory Generation Using Curvature Segments," which are incorporated by reference herein in their entireties for all purposes.

[0027] In various examples, the active prediction component 104 may also, or instead, determine the output data 108 based at least in part on an occupancy grid including a plurality of grid points representing pixels (e.g., discretized portions) in the environment. As a non-limiting example, the active prediction component 104 may output one or more occupancy grids corresponding to one or more times in the future. A predicted intersection may be determined if a grid portion overlaps with a known or expected position of the vehicle at that future time. In various examples, the active prediction component 104 may process speed, position, trajectory, or other information associated with the object and the vehicle to determine a response to the simulation by the vehicle computing device (and / or vehicle safety system). In various examples, the occupancy grids and potential intersections determined by the active prediction component 104 may be from different heads of the same neural network.

[0028] The output data 108 from the active prediction component 104 may be used by the vehicle computing device in a variety of ways. For example, the active prediction component 104, in conjunction with a planning component of the vehicle computing device (e.g., planning component 426), may determine to generate a signal indicative of information related to the output data 108 (e.g., a cost indicative of a potential intersection) to control the vehicle 102 within the environment 100 (e.g., to determine the vehicle trajectory and / or control the propulsion system, braking system, or steering system). The output data 108 may also, or instead, be used to run a simulation by setting conditions (e.g., intersections, number of objects, likelihood of objects exhibiting anomalous behavior, etc.) for use during the simulation (e.g., by the vehicle computing device or a computing device remote from the vehicle) to predict movement associated with objects within the environment.

[0029] The training component 450 of the computing device 436 and / or a training component of a remote computing device, such as the vehicle computing device 404, may be implemented to train the active prediction component 104. The training data may include a wide variety of data, such as image data, video data, lidar data, radar data, audio data, other sensor data, etc., associated with known or desired values ​​(e.g., desired classifications, inferences, predictions, etc.). Such values ​​may be generally referred to as “ground truth.” By way of example, the training data may be used for image classification and thus may include images of an environment captured by an autonomous vehicle and associated with one or more classifications. In some examples, such classifications may be based on user input (e.g., user input indicating that an image shows a particular type of object) or may be based on the output of another machine learning model. In some examples, such labeled classifications (or more generally, labeled outputs associated with the training data) may be referred to as ground truth.

[0030] 2 is a diagram of another example environment 200 in which one or more models determine potential interactions between one or more objects and a vehicle at a future time. For example, computing device 202 may implement active prediction component 104 to determine output data 108 representing potential interactions (e.g., intersections or nearby intersections (e.g., within a threshold distance)) between one or more objects (e.g., object 110, object 112, and / or object 114) and vehicle 102. In some examples, computing device 202 may be associated with vehicle computing device 404 and / or computing device 436.

[0031] 2 , the active prediction component 104 comprises an initialization component 204, which further comprises a memory allocation component 206 and an interaction component 208. In various examples, the initialization component 204 performs functions to receive input data 106 and perform “pre-processing” of the input data 106. For example, the initialization component 204 may receive state data indicative of an environment, objects, and a vehicle (e.g., the vehicle 102), sensor data, etc., and implement the memory allocation component 206 to allocate the input data 106 to one or more memories associated with the computing device 202. In various examples, allocating or allocating data in memory includes the memory allocation component 206 determining locations or portions of memory for efficient storage and allocation of the input data 106. In this manner, a processor (e.g., one or more GPUs) of the computing device 202 can efficiently process the input data 106 during generation of the output data 108. In some examples, the memory allocation component 206 receives the input data 106 from one or more CPUs and allocates the input data 106 to one or more GPUs.

[0032] The interaction component 208 receives input data 106 from one or more memories after being allocated by the memory allocation component 206 and provides functionality to determine potential interactions between two objects in the environment and / or one object and the vehicle at a future time. For example, the interaction component 208 can process trajectories 122(1), 122(2), and 122(3) associated with the vehicle 102 and trajectories 118(1), 118(2), and 118(3) associated with the object 112. In some examples, additional object trajectories up to 118(N) (N is an integer greater than 1) can also be included. In some examples, the trajectories 122(1), 122(2), and 122(3) can be associated with actions such as changing lanes or turning left (relative to the vehicle 102), going straight, and changing lanes or turning right, respectively. In some examples, trajectory 118(1) can be associated with a first velocity, trajectory 118(2) can be associated with a second velocity that is less than the first velocity, and trajectory 118(3) can be associated with a third velocity that is less than the second velocity. Although three actions are illustrated for vehicle 102 and three trajectories are illustrated for object 112, any number of action and / or trajectory types can be determined or received by active prediction component 104. Additionally, while FIG. 2 illustrates an interaction between vehicle 102 and object 112, techniques can include determining an interaction between two objects.

[0033] Additionally or alternatively, in various examples, the interaction component 208 can determine interactions between the vehicle 110, the vehicle 112, and the pedestrian 114, just to name a few. In various examples, multiple combinations of potential intersections can be determined between a first set of trajectories and a second set of trajectories associated with the object and / or vehicle.

[0034] FIG. 2 shows trajectory 118(1) associated with vehicle 112 intersecting vehicle trajectory 122(1) at intersection 210, and trajectory 118(1) intersecting vehicle trajectory 122(2) at intersection 212 and intersection 214. Also shown in FIG. 2, trajectory 118(1) associated with vehicle 112 intersects vehicle trajectory 122(3) at intersection 216, intersection 218, and intersection 220. FIG. 2 further illustrates areas within environment 200 where trajectory 118(1) does not intersect vehicle trajectory 122(1), vehicle trajectory 122(2), or vehicle trajectory 122(3). In some examples, potential intersections can be determined for object trajectory 118(1) and / or for other objects and their associated trajectories.

[0035] Generally, in various examples, the active prediction component 104 can determine, based at least in part on output by the interaction component 208, intersections at which objects (e.g., object 110, object 112, and / or object 114) may interact with the vehicle 102. In such examples, the active prediction component 104 can further process object trajectories associated with potential intersections (e.g., intersections 210, 212, 214, 216, 218, and 220), as described in connection with FIG. 3 . In other examples, object trajectories associated with intersections that are most likely to initially interact with the vehicle 102 can be selected by the active prediction component 104 for further processing. In some examples, the active prediction component 104 can make an initial determination that interaction is likely not possible, thereby conserving computational resources from processing additional details associated with such intersections (e.g., running simulations, determining estimated states of the objects or vehicle, etc.).

[0036] In various examples, one or more of vehicle trajectories 122(1), 122(2), 122(3) and / or object trajectories (e.g., trajectories 116(1), 116(2), 118(1), 118(2)... 118(N), and 120) may represent a "corridor" as opposed to the lines shown in FIG. 2. For example, any of the trajectories as described herein may represent an area of ​​environment 200 that may be occupied by a vehicle and / or object as the vehicle traverses the environment.

[0037] In some examples, the active prediction component 104 may associate one or more of the intersection points 210, 212, 214, 216, 218, and 220 with an intersection probability and output the intersection point associated with the object that is most likely to reach the intersection point first, such as by a simulation component as illustrated in FIG. 3. In some examples, two objects (e.g., vehicle 110 and vehicle 112) may intersect with the same vehicle trajectory, and the active prediction component 104 may identify or determine which of the two objects (or their associated object trajectories) to process with the simulation component based at least in part on control policies associated with rules of the road, right-of-way logic, physics, kinematics, dynamics, etc. In this manner, computational resources may be saved, for example, associated with processing data associated with a scenario in which an object arrives at an intersection with vehicle 102 after another object. That is, initialization component 204 may select one or more object trajectories (e.g., trajectories 116(1), 116(2), 118(1), 118(2), and 120) for further processing based at least in part on which of the trajectories reaches its respective intersection with vehicle 102 first. In various examples, initialization component 204 may select a trajectory from among the multiple trajectories based at least in part on a tree search that includes logic for identifying which objects are prioritized for potential interaction with the vehicle.

[0038] In some examples, the tree search can determine potential interactions between object trajectories of different objects and / or between the object trajectories and the vehicle trajectory, and the potential interactions determined by the tree search can be used in a simulation, as further described in FIG. 3 and elsewhere. In various examples, the tree search can identify potential interactions over time to reduce the amount of potential interactions at later times (e.g., every second or other time interval, the tree search can determine the most likely interactions between the object and the vehicle). In some examples, vehicle actions can be explored as various branches of the tree search, and tree branches can be pruned or ignored if the cost associated with the action meets or exceeds a threshold cost. As a non-limiting example, one or more actions can be contemplated at future discrete steps (e.g., fixed time, distance intervals, or some other event-based step). In such examples, the tree can branch at such discrete points based on different actions the vehicle can take at those points, and the methods described herein can be used to select between those branches when expanding the tree. For example, a branch can be explored that has the lowest cost and / or is free of adverse events (e.g., collisions, jarring control, etc.).

[0039] 3 is an illustration of another example environment 300 in which one or more models determine potential vehicle and / or object states at future points in time. For example, computing device 202 may implement active prediction component 104 to determine output data 108 representing potential future states for one or more objects (e.g., object 110, object 112, and / or object 114) and / or vehicle 102. In some examples, computing device 202 may be associated with vehicle computing device 404 and / or computing device 436.

[0040] 3 illustrates the active prediction component 104 with a simulation component 302 that further includes a control policy 304 and an integration component 306. FIG. 3 further illustrates the active prediction component 104 with a cost determination component 308.

[0041] In some examples, the simulation component 302 can provide functionality to simulate vehicle actions 310 and 312 at different times in the future. Each of the vehicle actions 310 and 312 can represent a change in vehicle state (e.g., a change in speed, acceleration, rotation rate, yaw, etc.) from a first time to a second time. In some examples, an example vehicle action can include not changing the vehicle state (e.g., staying in place, such as staying within a lane, maintaining the same steering action, braking action, and / or acceleration action). In some examples, the simulation component 302 can initiate an example scenario representing changes in vehicle state over time. In such examples, the scenario can include predictions regarding the future position and / or velocity of objects in the environment proximate to the vehicle relative to the node 210.

[0042] In some examples, the simulation component 302 can simulate possible actions for the vehicle and / or object over time. For example, if the vehicle takes a first action at a first time, such as one second into the simulation, the simulation component 302 can determine a second action for the vehicle based on the first action taken by the vehicle. In this way, the simulation component 302 can provide the ability to "actively" consider possible actions at each instance of the simulation (e.g., over four seconds), similar to how a driver makes decisions when operating a vehicle. In some examples, the simulation component 302 can determine actions for both the vehicle 102 and the object 112 that depend on previous actions as time increases during the simulation.

[0043] The control policies 304 may include one or more of a physics policy, a dynamics policy, a kinematics policy, and / or a rule policy that indicate how vehicles and objects can potentially interact or that indicate information about rules of the road, such as rights of way, associated with a road, intersection, or navigable surface. In some examples, the active prediction component 104 may implement a machine learning model that accesses the control policies 304 to determine actions, trajectories, positions, or other data associated with vehicles and / or objects. For example, the control policies 304 may be used by the machine learning model during a simulation performed by the simulation component 302.

[0044] As the vehicle 102 navigates within the environment 300, the vehicle 112 may approach the vehicle 102 such that the active prediction component 104 determines different potential actions (e.g., braking actions, accelerating actions, etc.) for the vehicle 102 to avoid a collision with the vehicle 102. While FIG. 3 shows the simulation component 302 representing vehicle actions 310 and 312 for the vehicle 112, additional objects (whether associated with an intersection or not) may also or instead be processed by the simulation component 302.

[0045] As shown in FIG. 3 , the output of the simulation component 302 may include predicted scenarios at times T0, T1, ... T N (where N is an integer greater than 1) that indicate future states of the environment. For example, sensor data associated with the vehicle 102 may be used to run scenarios to identify potential objects that may intersect with the vehicle 102 at different times in the future. In one particular example, the scenario may run for four seconds into the future at one-second intervals to identify objects in the area around the vehicle 102, including the vehicle 112. As shown in FIG. 3 , as time changes from time T0 to time T1, the vehicle 112 and the vehicle 102 are approaching each other. At time T1, the simulation component 302 may determine a vehicle action 310 based at least in part on predicting an object action 314 (e.g., a change in speed, acceleration, braking, etc.) associated with the vehicle 112. For example, the vehicle action 310 may include steering, accelerating, and / or braking to cause the vehicle 112 to avoid the object action 314. At time T2, the simulation component 302 can determine a vehicle action 312 based at least in part on predicting an object action 316 associated with the vehicle 112 .

[0046] In various examples, vehicle action 310 and vehicle action 312 can be determined at least in part based on accessing and / or storing data in memory (e.g., by memory allocation component 206) for processing by one or more GPUs. As time changes in the simulation, object state from a first time (e.g., time T1) can be accessed for consideration during another time (e.g., time T2) to minimize the amount of sensor data stored in memory, thereby enabling parallel execution in an efficient manner (e.g., as opposed to not implementing memory allocation component 206). In some examples, memory allocation component 206 can dynamically adjust the size of storage memory during time intervals during a scenario (e.g., during each second of a 4-second simulation) to reduce the number of registers per thread and / or reduce the number of times memory is accessed by components of active prediction component 104.

[0047] In various examples, vehicle action 310, vehicle action 312, object action 314, and / or object action 316 can be determined based at least in part on control policy 304 and integration component 306. For example, integration component 306 can predict a response by vehicle 102 to a scenario at time T0, a scenario at time T1, and / or a scenario at time Tn based at least in part on data associated with control policy 304 (e.g., information indicating rules of the road, such as rights-of-way associated with a road, intersection, or navigable surface). In such examples, integration component 306 can generate a signal indicative of the response and send the signal to another model or component of the vehicle computing device to validate a command, such as a decision by planning component 426 (e.g., determining a candidate trajectory for vehicle 102).

[0048] In some examples, actions by the vehicle and / or object may change from a first time to a second time during the simulation.

[0049] In various examples, the cost determination component 308 can determine costs associated with the vehicle actions and / or object actions determined by the simulation component 302. For example, the cost determination component 308 can determine a baseline cost, an obstacle cost, a steering cost, an acceleration cost, etc. associated with the vehicle actions 310 and 312 and / or the object actions 314 and 316.

[0050] In some examples, the cost determination component 308 may determine a cost associated with each of the aforementioned actions (also referred to as one associated with a reference action). For example, the cost determination component 308 may determine a first obstacle cost and a second obstacle cost that represent distance relative to the vehicle 102 and identify objects having a higher cost associated with being more likely to intersect with the vehicle 102. Additionally or alternatively, the cost determination component 308 may determine a reference cost of the vehicle 102 relative to an available roadway centerline and identify an amount of deviation from the centerline that may avoid the object. In various examples, costs associated with or otherwise determined by the cost determination component 308 may be used to select a candidate trajectory for the vehicle 102 (e.g., the planning component may receive a signal indicating the lowest aggregated cost that represents the safest potential action). Additional examples of vehicle architectures that determine costs are described, for example, in U.S. patent application Ser. No. 16 / 147,492, filed Sep. 28, 2018, entitled "Trajectory Generation and Optimization Using Closed-Form Numerical Integration in Route-Relative Coordinates," which is incorporated herein by reference in its entirety for all purposes. Additional examples of path planning in environments are described, for example, in U.S. patent application Ser. No. 16 / 179,679, filed Nov. 2, 2018, entitled "Adaptive Scaling in Trajectory Generation," which is incorporated herein by reference in its entirety for all purposes.

[0051] Although described as separate systems, in some examples, the techniques for evaluating a trajectory described herein with respect to Figures 1-3 may be implemented by other vehicle systems, components, and / or computing devices. For example, as described in further detail with respect to Figure 4, the prediction techniques described herein with respect to Figures 1-3 may be implemented at least in part by or in association with the perception component, planning component, and / or model component in Figure 4.

[0052] 4 is a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, the system 400 may include a vehicle, such as a vehicle 402.

[0053] The vehicle 402 may include a vehicle computing device 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, and one or more drive systems 414.

[0054] Vehicle computing device 404 may include one or more processors 416 and memory 418 communicatively coupled to the one or more processors 416. In the illustrated example, vehicle 402 is an autonomous vehicle, but vehicle 402 may be any other type of vehicle, such as a semi-autonomous vehicle, or any other system having at least an image capture device (e.g., a camera-enabled smartphone). In some examples, autonomous vehicle 402 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the trip, and in which a driver (or passenger) is not expected to control the vehicle at any time. However, in other examples, autonomous vehicle 402 may be a fully or partially autonomous vehicle having other levels or classifications.

[0055] In various examples, the vehicle computing device 404 may store sensor data associated with the object's actual position at the end of the set of estimated states (e.g., the end of a time period) and may use this data as training data to train one or more models. In some examples, the vehicle computing device 404 may provide the data to a remote computing device (i.e., a computing device separate from the vehicle computing device, such as computing device 436) for data analysis. In such examples, the remote computing device may analyze the sensor data to determine the object's actual position, velocity, heading, etc. at the end of the set of estimated states. Additional details of training machine learning models based on stored sensor data by minimizing the difference between the actual position and the predicted position and / or predicted trajectory are described in U.S. Patent Application No. 16 / 282,201, filed March 12, 2019, entitled "Motion Prediction Based on Appearance," which is incorporated herein by reference.

[0056] In the depicted example, memory 418 of vehicle computing device 404 stores a localization component 420, a perception component 422, a prediction component 424, a planning component 426, one or more system controllers 428, one or more maps 430, and model components 432 including one or more models, such as a first model 434A, a second model 434B, through an Nth model 434N (collectively “models 434”), where N is an integer. While shown in FIG. 4 as residing in memory 418 for illustrative purposes, localization component 420, perception component 422, prediction component 424, planning component 426, one or more system controllers 428, one or more maps 430, and / or model components 432 including models 434 may additionally or alternatively be accessible to vehicle 402 (e.g., may be stored in or otherwise accessible in memory remote from vehicle 402, such as memory 440 of remote computing device 436).

[0057] In at least one example, the localization component 420 includes functionality to receive data from the sensor system 406 to determine the position and / or orientation of the vehicle 402 (e.g., one or more of x position, y position, z position, roll, pitch, or yaw). For example, the localization component 420 can include and / or request / receive a map of the environment, such as from the map 430 and / or the map component 446, and can continuously determine the position and / or orientation of the autonomous vehicle within the map. In some examples, the localization component 420 can utilize simultaneous localization and mapping (SLAM), calibration, localization, and mapping simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least-squares optimization, etc. to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position of the autonomous vehicle. In some examples, the localization component 420 may provide data to various components of the vehicle 402 to determine the initial position of the autonomous vehicle for determining the relevance of objects to the vehicle 402, as described herein.

[0058] In some examples, perception component 422 may include functionality to perform object detection, segmentation, and / or classification. In some examples, perception component 422 may provide processed sensor data indicative of the presence of an object (e.g., an entity) in proximity to vehicle 402 and / or the classification of the object as an object type (e.g., a motor vehicle, a pedestrian, a bicyclist, an animal, a structure, a tree, a road surface, a curb, a sidewalk, an unknown, etc.). In some examples, perception component 422 may provide processed sensor data indicative of the presence of a stationary entity in proximity to vehicle 402 and / or the classification of the object as a type of stationary entity (e.g., a building, a tree, a road surface, a curb, a sidewalk, an unknown, etc.). In additional or alternative examples, perception component 422 may provide processed sensor data indicative of one or more features associated with a detected object (e.g., a tracked object) and / or an environment in which the object is located. In some examples, characteristics associated with an object may include, but are not limited to, x position (global and / or local position), y position (global and / or local position), z position (global and / or local position), orientation (e.g., roll, pitch, yaw), object type (e.g., classification), object velocity, object acceleration, object range (size), etc. Characteristics associated with an environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, time of day, day of the week, season, weather conditions, darkness / light indicators, etc.

[0059] The prediction component 424 can generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 424 can generate one or more probability maps for vehicles, pedestrians, animals, etc. within a threshold distance from the vehicle 402. In some examples, the prediction component 424 can measure the trajectories of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories for the objects based on the observed and predicted behavior. In some examples, the one or more probability maps can represent the intent of one or more objects in the environment.

[0060] In some examples, the prediction component 424 may generate predicted trajectories for objects (e.g., objects) in the environment and / or may generate predicted candidate trajectories for the vehicle 402. For example, the prediction component 424 may generate one or more predicted trajectories for objects within a threshold distance from the vehicle 402. In some examples, the prediction component 424 may measure the trace of the object and generate a trajectory for the object based on the observations and predicted behavior.

[0061] In general, the planning component 426 can determine a path for the vehicle 402 to take to traverse an environment. For example, the planning component 426 can determine various routes and trajectories, as well as various levels of detail. For example, the planning component 426 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the path can include a series of waypoints for traveling between the two locations. As non-limiting examples, the waypoints can include roads, intersections, Global Positioning System (GPS) coordinates, etc. Furthermore, the planning component 426 can generate instructions for guiding the autonomous vehicle along at least a portion of the path from the first location to the second location. In at least one example, the planning component 426 can determine how to guide the autonomous vehicle from a first waypoint in a sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can be a candidate trajectory, or a portion of a trajectory. In some examples, multiple trajectories may be generated substantially simultaneously (e.g., within technical tolerances) according to receding horizon techniques. A single path of the multiple paths in the receding horizon data having the highest confidence level may be selected for maneuvering the vehicle. In various examples, the planning component 426 may select a trajectory for the vehicle 402 based at least in part on receiving data representing the output of the model component 432.

[0062] In other examples, the planning component 426 can alternatively or additionally use data from the localization component 420, the perception component 422, and / or the prediction component 424 to determine a path for the vehicle 402 to follow to traverse an environment. For example, the planning component 426 can receive data from the localization component 420, the perception component 422, and / or the prediction component 424 regarding objects associated with the environment. Using this data, the planning component 426 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location) and avoid objects in the environment. In at least some examples, such a planning component 426 can determine that no such collision-free path exists and then provide a path that leads the vehicle 402 to a safe stop that avoids all collisions and / or otherwise mitigates damage. Additionally or alternatively, the planning component 426 can determine a route for the vehicle 402 to follow based at least in part on data received from the simulation component 302 and / or the cost determination component 308, as described in FIG. 3 and elsewhere.

[0063] In at least one example, vehicle computing device 404 includes one or more system controllers 428, which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 402. System controllers 428 may communicate with and / or control corresponding systems of drive system 414 and / or other components of vehicle 402.

[0064] The memory 418 may further include one or more maps 430 that may be used by the vehicle 402 to navigate within the environment. For purposes of this description, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the general environment. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual “surfels” (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the vehicle 402 may be controlled based at least in part on the map 430. That is, the map 430 may be used in conjunction with the localization component 420, the perception component 422, the prediction component 424, and / or the planning component 426 to determine the position of the vehicle 402, detect objects in the environment, generate paths, and determine actions and / or trajectories for navigating the environment.

[0065] In some examples, one or more maps 430 may be stored on a remote computing device (such as computing device 436) accessible via network 442. In some examples, multiple maps 430 may be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.) Storing multiple maps 430 may have similar memory requirements but increase the speed at which data in the maps can be accessed.

[0066] 4, the vehicle computing device 404 may include a model component 432. The model component 432 may be configured to perform the functions of the active prediction component 104, including predicting the likelihood of object intersections, such as objects 110, 112, and 114 of FIG. 1. In various examples, the model component 432 may receive one or more features associated with detected objects from the perception component 422 and / or the sensor system 406. In some examples, the model component 432 may receive environmental characteristics (e.g., environmental factors, etc.) and / or weather characteristics (e.g., weather factors, such as snow, rain, ice, etc.) from the perception component 422 and / or the sensor system 406. Although shown separately in FIG. 4, the model component 432 may be part of the prediction component 424, the planning component 426, or other components of the vehicle 402.

[0067] In various examples, the model component 432 may send predictions from one or more models 434 that may be used by the prediction component 424 and / or the planning component 426 to generate one or more predicted trajectories (e.g., heading, speed, etc.) of the object, such as from the prediction component. In some examples, the planning component 426 may determine one or more actions (e.g., base actions and / or sub-actions) for the vehicle 402, such as candidate vehicle trajectories. In some examples, the model component 432 may be configured to determine whether an object intersects at an intersection based at least in part on the one or more actions of the vehicle 402. In some examples, the model component 432 may be configured to determine actions applicable to the environment based on environmental characteristics, weather characteristics, etc.

[0068] The model component 432 may generate a set of estimated states of the vehicle and one or more detected objects moving forward in the environment over a period of time. The model component 432 may generate a set of estimated states for each action (e.g., base action and / or sub-action) determined to be applicable to the environment. The set of estimated states may include one or more estimated states, each estimated state including an estimated position of the vehicle and an estimated position of the detected object. In some examples, the estimated state may include an estimated position of the detected object at an initial time (T=0) (e.g., the current time).

[0069] The estimated position may be determined based on a detected trajectory and / or a predicted trajectory associated with the object. In some examples, the estimated position may be determined based on assumptions of a substantially constant velocity and / or a substantially constant trajectory (e.g., little or no lateral movement of the object). In some examples, the estimated position (and / or potential trajectory) may be based on passive and / or active predictions. In some examples, the model component 432 may utilize physics- and / or geometry-based techniques, machine learning, linear temporal logic, tree search methods, heat maps, and / or other techniques to determine the predicted trajectory and / or estimated position of the object.

[0070] In various examples, the estimated states may be generated periodically throughout a time period. For example, the model component 432 may generate estimated states at 0.1 second intervals throughout a time period. In another example, the model component 432 may generate estimated states at 0.05 second intervals. The estimated states may be used by the planning component 426 in determining actions (e.g., vehicle action 310 and vehicle action 314) to be taken by the vehicle 402 within the environment.

[0071] In various examples, the model component 432 may utilize machine learning techniques to predict a risk associated with the evaluated trajectory. In such examples, a machine learning algorithm may be trained to determine, based on sensor data and / or previous predictions by the model, that an object is likely to behave in a particular way relative to the vehicle 402 at a particular time during a set of estimated states (e.g., time periods). In such examples, one or more of the vehicle 402 state (position, velocity, acceleration, trajectory, etc.) and / or the object state, classification, etc. may be input into such a machine learning model, and a behavior prediction may then be output by the model.

[0072] In various examples, the characteristics associated with each object type may be used by model component 432 to determine the velocity or acceleration of the object, which may be used to predict potential intersections between objects and / or between vehicle 402 and one or more objects. Examples of object type characteristics may include, but are not limited to, maximum longitudinal acceleration, maximum lateral acceleration, maximum vertical acceleration, maximum velocity, maximum change in direction for a given velocity, etc.

[0073] As can be appreciated, the components discussed herein (e.g., localization component 420, perception component 422, prediction component 424, planning component 426, one or more system controllers 428, one or more maps 430, model component 432 including model 434) are described as separated for illustrative purposes, however, the operations performed by the various components may be combined or performed in any other component.

[0074] Although examples are shown in which the techniques described herein are implemented by a planning component and / or a model component of a vehicle, in some examples, some or all of the techniques described herein may be implemented by another system of the vehicle, such as a secondary safety system. Generally, such an architecture may include a first computing device for controlling the vehicle 402 and a secondary safety system operating on the vehicle 402 to verify the operation of the primary system and control the vehicle 402 to avoid a collision.

[0075] In some examples, some or all aspects of the components discussed herein may include any model, technique, and / or machine learning technique. For example, in some examples, the components in memory 418 (and memory 440 described below) may be implemented as neural networks.

[0076] As described herein, an exemplary neural network is a technology that passes input data through successively connected layers to produce an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of technology in which output is generated based on learned parameters.

[0077] Although described in the context of neural networks, any type of machine learning may be used consistent with the present disclosure. For example, machine learning techniques may include regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based techniques (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), Elastic Net, least-angle regression (LARS)), decision tree techniques (e.g., classification and regression tree (CART), iterative dichotomizer 3 (ID3), chi-squared automated interaction detection (CHAID), decision strain, conditional decision tree), Bayesian techniques (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence algorithm (AODE)), and others. estimators), Bayesian confidence networks (BNNs), Bayesian networks), clustering techniques (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning techniques (e.g., perceptrons, backpropagation, Hopfield networks, RBFNs (Radial Basis Function Networks)Network), deep learning techniques (e.g., Deep Boltzmann Machines (DBM), Deep Confidence Networks (DBN), Convolutional Neural Networks (CNN), stacked autoencoders), dimensionality reduction techniques (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble techniques (e.g., Boosting, Bootstrap Aggregation (Bagging), AdaBoost, Hierarchical Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forests), SVM (Support Vector Machine), supervised learning, unsupervised learning, semi-supervised learning, etc. Further example architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, etc.

[0078] In at least one example, sensor system 406 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time-of-flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. Sensor system 406 may include multiple instances of each of these or other types of sensors. For example, lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of vehicle 402. As another example, camera sensors may include multiple cameras positioned at various locations about the exterior and / or interior of vehicle 402. Sensor system 406 may provide input to vehicle computing device 404. Additionally or alternatively, the sensor system 406 may transmit sensor data over one or more networks 442 to one or more computing devices 436 at a particular frequency, after a predetermined period of time, in near real time, etc.

[0079] The vehicle 402 may also include one or more emitters 408 for emitting light and / or sound. The emitters 408 may include interior audio and visual emitters to communicate with occupants of the vehicle 402. By way of non-limiting example, the interior emitters may include speakers, lighting, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). The emitters 408 may also include exterior emitters. By way of non-limiting example, the exterior emitters may include lights that indicate direction of travel or other indicators of the vehicle's actions (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may comprise acoustic beam steering technology.

[0080] Vehicle 402 may also include one or more communication connections 410 that enable communication between vehicle 402 and one or more other local or remote computing devices. For example, communication connection 410 may facilitate communication with other local computing devices on vehicle 402 and / or drive system 414. Communication connection 410 may also enable the vehicle to communicate with other nearby computing devices (e.g., remote computing device 436, other nearby vehicles, etc.) and / or one or more remote sensor systems 444 for receiving sensor data. Communication connection 410 may also enable vehicle 402 to communicate with remote teleoperation computing devices or other remote services.

[0081] The communication connection 410 may include a physical and / or logical interface for connecting the vehicle computing device 404 to another computing device or network, such as the network 442. For example, the communication connection 410 may enable Wi-Fi-based communications over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

[0082] In at least one example, the vehicle 402 may include one or more drive systems 414. In some examples, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems 414, the individual drive systems 414 may be located at opposite ends of the vehicle 402 (e.g., the front and rear, etc.). In at least one example, the drive system 414 may include one or more sensor systems to detect conditions surrounding the drive system 414 and / or the vehicle 402. By way of example, and not limitation, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors that acoustically detect objects in the vicinity of the drive module, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be intrinsic to the drive system 414. In some cases, a sensor system on drive system 414 may overlap or complement a corresponding system on vehicle 402 (e.g., sensor system 406).

[0083] The drive system 414 may include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electronic actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as DC / DC converters, a high-voltage junction, a high-voltage cable, a charging system, a charge port, etc.). Additionally, the drive system 414 may include a drive module controller, which may receive and preprocess data from sensor systems and control the operation of various vehicle systems. In some examples, the drive module controller may include one or more processors and memory communicatively coupled to the one or more processors. The memory may store one or more modules and perform various functions of the drive system 414. Additionally, the drive system 414 may also include one or more communication connections that enable each drive module to communicate with one or more other local or remote computing devices.

[0084] In at least one example, direct connection 412 may provide a physical interface to couple one or more drive systems 414 with the body of vehicle 402. For example, direct connection 412 may allow for the transfer of energy, fluid, air, data, etc. between drive system 414 and the vehicle. In some examples, direct connection 412 may further removably secure drive system 414 to the body of vehicle 402.

[0085] In at least one example, the localization component 420, the perception component 422, the prediction component 424, the planning component 426, the one or more system controllers 428, the one or more maps 430, and the model component 432 may process the sensor data as described above and may transmit their respective outputs to the computing device 436 via one or more networks 442. In at least one example, the localization component 420, the perception component 422, the prediction component 424, the planning component 426, the one or more system controllers 428, the one or more maps 430, and the model component 432 may transmit their respective outputs to the remote computing device 436 at a particular frequency, after a predetermined period of time, in near real time, etc.

[0086] In some examples, vehicle 402 can transmit sensor data to computing device 436 over network 442. In some examples, vehicle 402 can receive sensor data from computing device 436 and / or remote sensor system 444 over network 442. The sensor data can include raw sensor data and / or processed sensor data and / or representations of sensor data. In some examples, the sensor data (raw or processed) can be transmitted and / or received as one or more log files.

[0087] The computing device 436 may include a processor 438 and a memory 440 that stores a map component 446, a sensor data processing component 448, and a training component 450. In some examples, the map component 446 may include functionality for generating maps of various resolutions. In such examples, the map component 446 may transmit one or more maps to the vehicle computing device 404 for navigation purposes. In various examples, the sensor data processing component 448 may be configured to receive data from one or more remote sensors, such as the sensor system 406 and / or the remote sensor system 444. In some examples, the sensor data processing component 448 may be configured to process the data and transmit the processed sensor data to the vehicle computing device 404 for use by the model component 432 (e.g., the model 434). In some examples, the sensor data processing component 448 may be configured to transmit raw sensor data to the vehicle computing device 404.

[0088] In some examples, the training component 450 can include functionality for training a machine learning model and output an evaluation trajectory. For example, the training component 450 can receive sensor data representing an object traversing an environment for a period of 0.1 milliseconds, 1 second, 3 seconds, 5 seconds, 7 seconds, etc. At least a portion of the sensor data can be used as input for training the machine learning model.

[0089] In some examples, the training component 450 may be executed by the processor 438 to train a machine learning model based on training data. The training data may include a wide variety of data, such as sensor data, audio data, image data, map data, inertial data, vehicle state data, historical data (log data), or combinations thereof, associated with a value (e.g., a desired classification, inference, prediction, etc.). Such values ​​may be generally referred to as “ground truth.” To illustrate, the training data may be used to determine a risk associated with an evaluated trajectory and thus may include data representing the environment captured by the autonomous vehicle and associated with one or more classifications or decisions. In some examples, such classifications may be based on user input (e.g., a user input indicating that the data indicates a particular risk) or the output of another machine learning model. In some examples, such labeled classifications (or more generally, labeled outputs associated with the training data) may be referred to as ground truth.

[0090] In some examples, the training component 450 can include functionality for training a machine learning model and output classification values. For example, the training component 450 can receive data representing labeled crash data (e.g., publicly available data, sensor data, and / or a combination thereof). At least a portion of the data can be used as input for training the machine learning model. Thus, by providing data of a vehicle traversing an environment, the training component 450 can be trained and output potential intersection points associated with objects, as described herein.

[0091] In some examples, training component 450 can include training data generated by a simulator. For example, the simulated training data can represent instances in which a vehicle collides with or nearly collides with objects in the environment to provide additional training examples.

[0092] Processor 416 of vehicle 402 and processor 438 of computing device 436 may be any suitable processor capable of executing instructions and may process data to perform operations as described herein. By way of non-limiting example, processors 416 and 438 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0093] Memory 418 and memory 440 are examples of non-transitory computer-readable media. Memory 418 and 440 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functionality attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples relevant to the description herein.

[0094] 4 is shown as a distributed system, it should be noted that in alternative examples, components of vehicle 402 may be associated with computing device 436 and / or components of computing device 436 may be associated with vehicle 402. That is, vehicle 402 may perform one or more functions associated with computing device 436, and vice versa.

[0095] 5A is a first portion of a flowchart illustrating an example process 500 for determining potential intersection points using one or more example models. For example, some or all of process 500 can be performed by one or more components in FIG. 4 as described herein. For example, some or all of process 500 can be performed by vehicle computing device 404 or computing device 202.

[0096] At operation 502, the process may include receiving a first set of predicted trajectories associated with a first object and a second set of trajectories associated with a second object. In some examples, operation 502 may include a vehicle computing device capturing sensor data from a lidar sensor, a radar sensor, etc. of the vehicle 402. In some examples, the sensor data may be processed to determine a two-dimensional representation of the environment (e.g., top-down multi-channel data, vector data, an occupancy grid, etc.). The sensor data may be received from one or more sensors on the vehicle and / or from one or more remote sensors. In some examples, one or more trajectories may be predicted by the vehicle computing device (e.g., prediction component 424) for each object detected in the environment surrounding the vehicle. The first set of predicted trajectories and / or the second set of predicted trajectories may be determined, for example, by the prediction component 424.

[0097] Operation 502 may, in some examples, include determining that first and second predicted trajectories associated with one or more objects have been received from the CPU, and allocating the first and second predicted trajectories to one or more GPUs by memory allocation component 206.

[0098] At operation 504, the process may include determining a predicted intersection point based on the first predicted trajectory of the first set and the second trajectory of the second set. In some examples, operation 504 may include a vehicle computing device implementing an initialization component 204 (or a different model), where the initialization component 204 is configured to predict, estimate, extrapolate, or otherwise determine whether an object trajectory will intersect or interact with the vehicle's trajectory. In various examples, a planned trajectory for the vehicle 102 may be received from the planning component 426 of the vehicle computing device 404 for use in determining interactions between each detected object and the vehicle at various intersection points representing each object and the vehicle's potential future positions. For example, the interaction component 208 may determine intersection points 210, 212, 214, 216, 218, and / or 220. In various examples, the initialization component 204 can use the input data 106 (e.g., object state data associated with each object) to identify a first object (e.g., object 112) that is most likely to reach an intersection with the vehicle 102 before a second object (e.g., object 110).

[0099] In some examples, the predicted intersection points can be determined based at least in part on grid points associated with one or more occupancy grids (e.g., grid 222). For example, input data 106 can be processed by active prediction component 104 to determine whether an object in the environment is associated with one of the grid points of the occupancy grid, and future potential interactions can be determined based on objects occupying the same grid points of the occupancy grid.

[0100] In operation 506, the process may include simulating the vehicle to perform a first action and a second action of the vehicle relative to at least a first predicted trajectory associated with the first object and a second trajectory associated with the second object. In some examples, operation 506 may include determining vehicle actions 310 and 312 representing acceleration actions, braking actions, steering actions, etc. relative to the first predicted trajectory and the second predicted trajectory. The first action and / or the second action may represent possible states of the vehicle at different times in the future.

[0101] At operation 508, the process may include determining an updated predicted trajectory associated with at least one of the first object or the second object based on the predicted intersection and the control policy. In some examples, operation 508 may include comparing a magnitude and direction of a velocity or acceleration associated with the object with a magnitude and direction of a velocity or acceleration associated with the vehicle. In various examples, the active prediction component 104 may implement a simulation component 302 including one or more machine learning models to determine which of one or more object trajectories will intersect with the vehicle trajectory and update the object trajectories based on a control policy including logic for determining rules of the road for objects in the environment and the vehicle (e.g., which objects have the right-of-way and which objects respond to traffic signals, speed bumps, etc.).

[0102] FIG. 5B is a second portion of a flowchart illustrating an example process for determining potential intersections using one or more example models.

[0103] At operation 510, the process may include determining a first cost associated with the first action and a second cost associated with the second action. In some examples, operation 510 may include the cost determination component 308 determining a baseline cost, an obstacle cost, an acceleration cost, and / or a steering cost for each baseline action. In some examples, the cost determination component 308 may also determine costs associated with object actions in a scenario implemented by the simulation component 302. In various examples, actions by the vehicle and / or object may include a braking action to brake the vehicle, a steering action to steer the vehicle, or an accelerating action to accelerate the vehicle, just to name a few.

[0104] At operation 512, the process may include determining a candidate trajectory for the vehicle based at least in part on the updated predicted trajectory, the first cost, and the second cost. In some examples, operation 512 may include the active prediction component 104 generating a signal indicating a potential interaction and associated cost for the vehicle to avoid a potential interaction with an object in the environment. In various examples, the active prediction component 104 may send a signal to other components of the vehicle computing device 404, such as the planning component 426, which causes the planning component 426 to generate candidate trajectories for the vehicle to consider when navigating within the environment. In some examples, operation 512 may include the planning component 426 utilizing information associated with the first cost and the second cost to determine a candidate trajectory or other action for the vehicle to avoid a potential intersection between the object and the vehicle.

[0105] At operation 514, the process may include controlling the vehicle in the environment based at least in part on the candidate trajectory. In some examples, operation 514 may include controlling the vehicle in the future environment by the vehicle computing device 404 based at least on information from the active prediction component 104. In various examples, the determination of a potential intersection by the active prediction component 104 can be verified (e.g., to occur or not occur at a future time) by processing additional input data representing the object and the environment over time.

[0106] In various examples, process 500 may return to 502 after performing operation 514. In such examples, the vehicle may continuously monitor for potential collisions and update and revise its decision regarding whether to activate safety systems (which, at least in some examples, may include performing one or more operations to mitigate or minimize the impact). In any of the examples described herein, the process may be repeated at a given frequency to generate one or more occupancy grids associated with one or more future times in order to make the above-described determinations.

[0107] 5A and 5B illustrate exemplary processes according to examples of the present disclosure. These processes are illustrated as logical flow graphs, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the software context, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed on one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order of the described operations is not intended to be construed as a limitation, and any number of the described operations can be omitted and / or combined in any order and / or in parallel to implement the process.

[0108] The methods described herein represent sequences of operations that can be implemented in hardware, software, or a combination thereof. In the software context, blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations described above. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be limiting, and any number of the described operations can be combined in any order and / or in parallel to implement a process. In some examples, one or more operations of a method may be omitted entirely. For example, an operation may include determining a first action and a second action by a vehicle on a selected trajectory without determining a respective cost of one or more of the vehicle actions. Furthermore, the methods described herein may be combined, in whole or in part, with each other or with other methods.

[0109] The various techniques described herein may be implemented in the context of computer-executable instructions or software, such as program modules, stored in computer-readable storage and executed by processors of one or more computing devices, such as those illustrated in the figures. Generally, program modules include routines, programs, objects, components, data structures, etc., that define operational logic for performing particular tasks and implement particular abstract data types.

[0110] Other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Additionally, although a specific division of responsibilities is defined above for purposes of discussion, the various functions and responsibilities may be distributed and divided in different manners, depending on the circumstances.

[0111] Similarly, software may be stored and distributed in a variety of ways and using different means, and the particular software storage and execution configurations described above can be varied in many different ways. Thus, software implementing the above techniques is not limited to the specifically described forms of memory, but can be distributed on various types of computer-readable media.

[0112] Example clauses Any of the example clauses in this section may be used with any of the example clauses and / or any of the other examples or embodiments described herein.

[0117] A: A system comprising one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to perform operations including receiving a first set of predicted trajectories associated with a first object and a second set of trajectories associated with a second object; determining a predicted intersection based on a first predicted trajectory of the first set and a second trajectory of the second set; simulating a vehicle to perform a first action and a second action on at least the first predicted trajectory associated with the first object and the second trajectory associated with the second object; determining an updated predicted trajectory associated with at least one of the first object or the second object based on the predicted intersection and a control policy; determining a first cost associated with the first action and a second cost associated with the second action; determining a candidate trajectory for the vehicle based at least in part on the updated predicted trajectory, the first cost, and the second cost; and controlling the vehicle in an environment based at least in part on the candidate trajectory.

[0113] B: The system described in paragraph A, wherein the control policy indicates at least one of a physics policy, a dynamics policy, a kinematics policy, or a right-of-way policy, and the right-of-way policy is associated with one or more of a road, an intersection, or a surface navigable by the vehicle.

[0114] C: The system described in either paragraph A or B, wherein the operations further include receiving map data of the environment and further determining the updated predicted trajectory based at least in part on the map data.

[0115] D: The system described in any one of paragraphs A to C, wherein the operations further include: determining an occupancy grid including a plurality of grid points representing a discretized portion of the environment based at least in part on the first predicted trajectory and the second trajectory; and determining the predicted intersection point based at least in part on a first occupancy of the first object and a second occupancy of the second object at grid points of the occupancy grid.

[0116] E: The system of any one of paragraphs A-D, wherein the operations further include inputting the first set of predicted trajectories and the second set of trajectories to a parallel processing unit, and determining the predicted intersection point based at least in part on output by the parallel processing unit.

[0117] F: A method comprising: receiving a first set of trajectories associated with a vehicle and a second set of predicted trajectories associated with an object; determining a predicted intersection based on a first trajectory of the first set and a second predicted trajectory of the second set; simulating actions to be performed by the vehicle on the second predicted trajectory associated with the object; determining an updated predicted trajectory associated with the object based on the predicted intersection and a control policy; and determining candidate trajectories for the vehicle based at least in part on the actions and the updated predicted trajectories.

[0118] G. The method of paragraph F, further comprising: determining a cost associated with the action; and controlling the vehicle within an environment based at least in part on the cost.

[0119] The method of either paragraph F or G, further comprising: H: determining an action for the object to perform on the first trajectory associated with the vehicle.

[0120] I: The method of any one of paragraphs F-H, further comprising: determining an occupancy grid including a plurality of grid points representing a discretized portion of an environment; and determining the predicted intersection between the first trajectory and the second predicted trajectory based at least in part on a first occupancy of the vehicle and a second occupancy of the object at grid points of the occupancy grid.

[0121] J: The method of any one of paragraphs F-I, wherein determining the predicted intersection is based at least in part on state data associated with one or more of the vehicle or the object.

[0122] K. The method of any one of paragraphs F-J, further comprising: receiving map data of the vehicle's environment; and determining, by a machine learning model, the updated predicted trajectory further based at least in part on the map data.

[0123] L: The method of any one of paragraphs F-K, wherein the control policy indicates rules associated with one or more of a road, an intersection, or a surface navigable by the vehicle.

[0124] M: The method of any one of paragraphs F-L, wherein the step of determining the updated predicted trajectory is further based at least in part on a tree search.

[0125] N: The method of any one of paragraphs F-M, further comprising: determining a plurality of combinations between a first set of trajectories associated with the vehicle and a second set of predicted trajectories associated with the object; and inputting the plurality of combinations into a parallel processing unit for processing.

[0126] O: The method of any one of paragraphs F-N, wherein the action is a first action associated with a first time, and further comprising the step of simulating a second action for the vehicle at a second time after the first time based at least in part on the first action and the control policy.

[0127] P: The method of any one of paragraphs F-O, further comprising: determining that the object is a first object and that the first object is associated with the predicted intersection at a first time; determining that a second object is associated with the predicted intersection at a second time after the first time; and determining the updated predicted trajectory associated with the first object or the second object based at least in part on the second object associated with the predicted intersection at the second time.

[0128] Q: One or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to perform operations including receiving a first set of trajectories associated with a vehicle and a second set of predicted trajectories associated with an object; determining a predicted intersection based on a first trajectory of the first set and a second predicted trajectory of the second set; simulating actions to be performed by the vehicle on the second predicted trajectory associated with the object; determining an updated predicted trajectory associated with the object based on the predicted intersection and a control policy; and determining a candidate trajectory for the vehicle based at least in part on the actions and the updated predicted trajectories.

[0129] R: The non-transitory computer-readable medium of paragraph Q, further comprising: determining a cost associated with the action; and controlling the vehicle within an environment based at least in part on the cost.

[0130] S: The non-transitory computer-readable medium of either paragraph Q or R, wherein the operations further include determining a plurality of combinations between a first set of trajectories associated with the vehicle and a second set of predicted trajectories associated with the object, and inputting the plurality of combinations to a parallel processing unit for processing.

[0131] T: The non-transitory computer-readable medium of any of paragraphs Q-S, wherein the control policy indicates at least one of a physics policy, a dynamics policy, a kinematics policy, or a right-of-way policy, and the right-of-way policy is associated with one or more of a road, an intersection, or a surface navigable by the vehicle.

[0132] Although the example clauses described above are described with respect to one particular implementation, it should be understood that in the context of this specification, the content of the example clauses may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Furthermore, any of Examples A-T may be implemented alone or in combination with one or more of the other Examples A-T.

[0133] summary One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0134] In the description of the examples, reference is made to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific examples of the claimed subject matter. It is understood that other examples may be used and modifications or substitutions, such as structural changes, may be made. Such examples, modifications, or substitutions do not necessarily depart from the intended scope of the claimed subject matter. While steps herein may be presented in a particular order, in some cases the order can be changed so that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. Disclosed procedures can also be performed in a different order. Furthermore, various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations can be readily implemented.

Claims

1. receiving a first set of trajectories associated with the vehicle and a second set of predicted trajectories associated with the object; determining a predicted intersection point based on a first trajectory of the first set and a second predicted trajectory of the second set; simulating one or more of a steering action, a braking action, and an accelerating action to be performed by the vehicle relative to the second predicted trajectory associated with the object; determining an updated predicted trajectory associated with the object based on the predicted intersection, a control policy, and a tree search, the tree search comprising: associating a vehicle operation of the vehicle with a branch of a decision tree; a deciding step including: ignoring a branch of the decision tree based on a cost associated with a particular action; determining a candidate trajectory for the vehicle based at least in part on the simulated actions and the updated predicted trajectory; A method comprising:

2. determining a cost associated with the simulated action; The method of claim 1 , further comprising controlling the vehicle within an environment based at least in part on the cost.

3. The method of claim 1 , further comprising: determining an action for the object to perform on the first trajectory associated with the vehicle.

4. determining an occupancy grid comprising a plurality of grid points representing a discretized portion of the environment; 2. The method of claim 1, further comprising: determining the predicted intersection between the first trajectory and the second predicted trajectory based at least in part on a first occupancy of the vehicle and a second occupancy of the object at grid points of the occupancy grid.

5. The method of claim 4 , wherein determining the predicted intersection is based at least in part on state data associated with one or more of the vehicle or the object.

6. receiving map data of the vehicle's environment; The method of claim 1 , further comprising: determining the updated predicted trajectory further based at least in part on the map data by a machine learning model.

7. The method of claim 1 , wherein the control policy indicates rules associated with one or more of a road, an intersection, or a surface navigable by the vehicle.

8. The method of claim 1 , wherein determining the updated predicted trajectory is further based at least in part on a cost associated with the particular maneuver meeting or exceeding a threshold cost.

9. determining a plurality of combinations between the first set of trajectories associated with the vehicle and the second set of predicted trajectories associated with the object; inputting the plurality of combinations into a parallel processing unit for processing; The method of claim 1 , further comprising: determining the predicted intersection point based at least in part on output by the parallel processing units.

10. the simulated action is a first action associated with a first time; simulating a second action for the vehicle at a second time after the first time based at least in part on the first action and the control policy.

11. the object is a first object, determining that the first object is associated with the predicted intersection point at a first time; determining, at a second time after the first time, that a second object is associated with the predicted intersection point; 2. The method of claim 1 , further comprising: determining the updated predicted trajectory associated with the first object or the second object based at least in part on the second object associated with the predicted intersection at the second time.

12. A computer program comprising coded instructions which, when executed on a computer, implements the method according to any of claims 1 to 11.

13. one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors; wherein the instructions, when executed, cause the system to: receiving a first set of trajectories associated with the vehicle and a second set of predicted trajectories associated with the object; determining a predicted intersection point based on a first trajectory of the first set and a second predicted trajectory of the second set; simulating one or more of a steering action, a braking action, and an accelerating action to be performed by the vehicle relative to the second predicted trajectory associated with the object; determining an updated predicted trajectory associated with the object based on the predicted intersection, a control policy, and a tree search, the tree search comprising: Associating a vehicle action of the vehicle with a branch of a decision tree; deciding, including ignoring a branch of the decision tree based on a cost associated with a particular action; determining a candidate trajectory for the vehicle based at least in part on the simulated actions and the updated predicted trajectory; A system that causes an operation including

14. determining a plurality of combinations between the first set of trajectories associated with the vehicle and the second set of predicted trajectories associated with the object; inputting the plurality of combinations into a parallel processing unit for processing; The system of claim 13 further comprising:

15. 15. The system of claim 13 or 14, wherein the control policy represents at least one of a physics policy, a dynamics policy, a kinematics policy, or a right-of-way policy, and the right-of-way policy is associated with one or more of a road, an intersection, or a surface navigable by the vehicle.

Images