Conditional object location prediction by machine learning models

Abstract

This paper describes a technique for predicting the location of objects in an environment. For example, the technique may include clustering the predicted object locations and using representative targets from one or more clusters to determine the predicted location of an autonomous vehicle. The clusters can vary in size, number of locations, etc. Object locations can be selected from one or more clusters for consideration during vehicle planning, which may include simulation.

Description

Related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 523,684, filed November 29, 2023, entitled “CONDITIONAL OBJECT POSITIONPREDICTION BY A MACHINE LEARNED MODEL,” the entire contents of which are incorporated herein by reference. Background Technology

[0002] Machine learning models can be used to predict the actions of various robotic devices. For example, planning systems in autonomous and semi-autonomous vehicles determine the actions the vehicle will take in its operating environment. Vehicle actions can be determined in part based on avoiding objects present in the environment. For example, actions such as yielding to pedestrians or changing lanes to avoid another vehicle in the road can be generated. Accurately predicting the future locations of objects in the environment helps in operating vehicles safely near those objects. Attached Figure Description

[0003] The accompanying drawings provide a detailed description. In the drawings, the leftmost numeral indicates the first drawing in which the reference numeral appears. The same reference numerals are used in different drawings to indicate similar or identical components or features.

[0004] Figure 1 This is an illustration of an autonomous vehicle in an example environment, where an example machine learning model processes input data to predict object locations.

[0005] Figure 2 An example block diagram of an example computer architecture for implementing the techniques described herein for generating example feature vectors is shown.

[0006] Figure 3 Another block diagram of an example computer architecture is shown for implementing the technique described herein for determining the location of example objects.

[0007] Figure 4A An example block diagram of a computing device is shown, illustrating an example prediction component for generating object location information.

[0008] Figure 4B An example block diagram of a computing device is shown, illustrating an example clustering component for generating object location information.

[0009] Figure 5 This is a block diagram of an example system used to implement the techniques described in this article.

[0010] Figure 6A This is the first part of a flowchart depicting an example process for determining the location of an object using one or more example models.

[0011] Figure 6B This is the second part of a flowchart depicting an example process for determining the location of an object using one or more example models. Detailed Implementation

[0012] This application describes techniques for predicting the location of objects in an environment. For example, a computing device can determine multiple locations or candidate locations corresponding to objects at different future times. In some examples, the computing device can determine the object's location based on the predicted location of an autonomous vehicle in the environment. For example, the computing device can predict the object's location based on how the autonomous vehicle behaves in response to the object moving to the predicted location. In some examples, the object's location determined by the computing device can be considered during vehicle planning, thereby improving vehicle safety when navigating an environment by planning for the probability that the object might occupy various locations.

[0013] In some instances, the computing device may implement a model (e.g., a prediction component) to predict object locations based on input data indicating where the autonomous vehicle might occupy in the future. For example, the model may receive the predicted location of the autonomous vehicle (e.g., from the prediction or planning component of the computing device) and determine output data representing the locations of one or more objects that could influence the autonomous vehicle during navigation to the predicted location. In some examples, the model may receive a single location of the autonomous vehicle in the next N seconds, where N is an integer. In other examples, the model may receive a trajectory of the autonomous vehicle indicating multiple locations along that trajectory for consideration in determining object locations.

[0014] The input data for this model may also include, or alternatively, object location data associated with multiple objects in the environment at previous times and / or within previous time periods. For example, the input data may indicate the historical object locations of dynamic objects (e.g., pedestrians, another vehicle, etc.) over time to capture different potential actions of the object, such as whether the object will perform a U-turn, left turn, remain stationary, change lanes, etc. The model may output a threshold number of object locations that collectively represent different potential actions of the object, even if the action is associated with a relatively low probability of occurrence (e.g., a U-turn is less likely than the object going straight). By outputting the threshold number of object locations as described herein, the model can determine output data capturing potential adverse behaviors of the object. As used herein, adverse behavior of an object means object behavior that affects or is likely to affect the operation of a vehicle, such as requiring the vehicle to move or change speed to avoid a collision or near miss (e.g., moving towards an autonomous vehicle, sudden or unstable actions, etc.).

[0015] In some examples, the object locations output by the model can be further processed by the same or different models to determine a subset of object locations. For example, the model can cluster the object locations output by the model based on the predicted location of the autonomous vehicle. For example, the model can determine a first cluster of at least some candidate locations from a set of candidate locations associated with the object that would cause a first behavior of the autonomous vehicle in future time, and a second cluster of at least some candidate locations from a set of candidate locations associated with the object that would cause a second vehicle behavior of the autonomous vehicle in future time. The first vehicle behavior and / or the second vehicle behavior can represent braking, steering, acceleration, or other actions of the autonomous vehicle. For example, clustering can be based on the vehicle performing a specific action in response to an object occupying any candidate location in the first cluster. That is, the model can determine candidate object locations to be included in the first or second cluster based on the vehicle's response to an object being in one of the candidate locations in the corresponding cluster (e.g., the vehicle can brake in response to an object in the first cluster, or change lanes in response to an object in the second cluster). Although only two clusters / behaviors are described in this example, in other examples, the model may output any number of clusters / behaviors depending on the characteristics of the environment (e.g., the number of lanes, the potential location of vehicles and / or objects, etc.). Further descriptions of cluster determination can be found throughout this disclosure, including with reference to the figures below.

[0016] The data output by the model can be used in various ways. For example, output data representing the location an object might occupy in the future can be transmitted to a computing device or component configured to control an autonomous vehicle. For instance, object locations (also known as candidate object locations) can be transmitted to a planning component of a vehicle computing device configured to determine planning data (e.g., vehicle trajectory, object trajectory, decision tree output, etc.). In some examples, the model's output data can be used to perform simulations, control the vehicle, verify or test the performance of the vehicle or its components, etc. As an example, and not a limitation, the model can determine object locations in a decision tree for controlling an autonomous vehicle (e.g., reference actions associated with the object locations can be included in the decision tree). The object locations output by the model can improve vehicle planning operations by implementing more realistic reference actions in the decision tree (e.g., planning for larger changes in potential object locations).

[0017] In some examples, one or more machine learning models may receive historical data indicating previous locations associated with one or more objects in an environment (e.g., a real-world or simulated environment) and output candidate object locations that the objects may occupy in the future. Machine learning models may also receive vehicle data indicating previous or predicted locations associated with autonomous vehicles in the environment as input data and output candidate vehicle locations that the autonomous vehicles may occupy in the future.

[0018] In various examples, the candidate object locations output by the model can be associated with clusters, datasets, distributions, etc. For example, the model can determine an object location cluster relative to one of the candidate vehicle locations based on the fact that each object location in the corresponding cluster causes the same vehicle action. For example, the model can assign multiple object locations to a cluster based on the fact that the vehicle action is the same for each candidate object location in the cluster at future times (e.g., when occupied by an object). Object locations can be determined independently of knowing the object's initial location or tracking the object to a candidate object location (thus saving otherwise associated computational resources). The initial location can, for example, represent the object's previous location before occupying a candidate location. In various examples, candidate locations can be determined based on the number of times an object has occupied various locations in the environment within a previous time period.

[0019] In some examples, the model's input data may include top-down representations (e.g., multiple layers or channels of an "image" representing data from a view of the driving surface from above) and / or feature vectors representing the environment (e.g., some embeddings or encodings representing the environment), objects, and / or the autonomous vehicle. In some examples, the computing device may receive sensor data, log data, map data, etc., as input and determine top-down representations and / or feature vectors representing objects, vehicles, and / or the environment. For example, a machine learning model (e.g., a graph neural network) may determine feature vectors based at least in part on input data representing object location, object trajectory, object state, vehicle information, simulation scenarios, real-world scenarios, etc. The computing device may receive feature vectors from the machine learning model as part of its input data. In various examples, feature vectors may be generated to represent the current state of an object (e.g., heading, speed, etc.) and / or the object's behavior over time (e.g., changes in the object's yaw, speed, or acceleration). In some examples, the machine learning model determines additional feature vectors to represent other objects and / or features of the environment.

[0020] In some examples, the machine learning model may receive a vector representation of data compiled into an image format representing a top-down view of the environment. The top-down view can be determined at least in part based on map data and / or sensor data captured from or associated with sensors of an autonomous vehicle in the environment. The vector representation of the top-down view may represent one or more of the following: object attributes (e.g., location, category, speed, acceleration, yaw, turn signal status, etc.), object history (e.g., location history, speed history, etc.), vehicle attributes (e.g., speed, location, etc.), pedestrian crossing permission, traffic light permission, etc. The data can be represented in the top-down view of the environment to capture the context of the autonomous vehicle (e.g., identifying the actions of other vehicles and pedestrians relative to that vehicle).

[0021] In some examples, the machine learning model may receive vector representations of data associated with one or more objects in the environment as input data. For example, the machine learning model may receive (or, in some examples, determine) one or more vectors representing one or more of the following associated with an object: position data, orientation data, heading data, velocity data, rate data, acceleration data, yaw rate data, or turning rate data.

[0022] In various examples, a computing device (e.g., a vehicle computing device) can be configured to determine the actions (e.g., trajectories for controlling the vehicle) that the vehicle should take during operation, based on the predicted object positions determined by one or more models. Actions can include reference actions (e.g., one of a set of maneuvers the vehicle is configured to perform in response to a dynamic operating environment), such as changing lanes to the right, changing lanes to the left, staying in the lane, and maneuvering around obstacles (e.g., double-parked vehicles, a group of pedestrians, etc.). Actions can further include sub-actions, such as speed changes (e.g., maintaining speed, accelerating, decelerating, etc.), position changes (e.g., changing position within the lane), etc. For example, actions can include staying in the lane (action) and adjusting the vehicle's position within the lane from a centered position to the left side of the lane (sub-action).

[0023] For each applicable action and sub-action, the vehicle computing system can implement different models and / or components to simulate future states (e.g., estimate states) by projecting the autonomous vehicle and related objects forward in the environment for a period of time (e.g., 5 seconds, 8 seconds, 12 seconds, etc.). The model can project objects based on their associated predicted trajectories (e.g., estimate the future position of objects). For example, the model can predict the vehicle's trajectory and predict attributes about the vehicle, including whether the vehicle will use the trajectory to reach the predicted location in the future. The vehicle computing device can project the vehicle forward based on the vehicle trajectory output by the model (e.g., estimate the vehicle's future position). The estimated state can represent the estimated position of the autonomous vehicle (e.g., estimated location) and the estimated position of related objects at a future time. In some examples, the vehicle computing device can determine relative data between the autonomous vehicle and objects in the estimated state. In such examples, relative data can include distance, position, speed, direction of travel, and / or other factors between the autonomous vehicle and objects. In various examples, the vehicle computing device can determine the estimated state at a predetermined rate (e.g., 10 Hz, 20 Hz, 50 Hz, etc.). In some examples, the rate at which the estimated state is determined can vary over time and / or based on one or more conditions (e.g., vehicle speed, the speed of objects in the environment, the number of objects in the environment, the type of operating driving domain (e.g., residential street vs. highway), whether the vehicle is occupied, etc.). In at least one example, the estimated state can be performed at a rate of 10 Hz (e.g., 80 estimated intentions over an 8-second time period).

[0024] In various examples, a vehicle computing system can store sensor data associated with the actual location of an object at the end of an estimated set of states (e.g., the end of a time period) and use this data as training data to train one or more models. For example, the stored sensor data (or perception data derived from it) can be retrieved by the model and used as input data to identify cues for recognizing objects (e.g., identifying the object's location, features, attributes, or pose). Such training data can be determined based on manual annotations and / or by identifying semantic information associated with changes in the object's location and / or orientation between times in the stored data. Furthermore, the locations detected within such time periods and associated with the object can be used to determine the true location to be associated with the object.

[0025] In some examples, the vehicle computing device can provide data such as log data, sensor data, and training data to a remote computing device (i.e., a computing device separate from the vehicle computing device) for data analysis. In such examples, the remote computing device can analyze the data to determine one or more labels for an image, the actual position of an object at the end of the estimated state set, yaw, velocity, acceleration, direction of travel, etc. In some such examples, real-world data can be associated with one or more of the following of the object represented in the stored data: position, trajectory, acceleration, and / or orientation. Real-world data can be determined (either manually labeled or determined by another machine learning model), and this real-world data can be used to determine the object's position. In some examples, the corresponding data can be input into a model to determine the output, and the difference between the determined output and the object's actual movement (or actual position data) can be used to train the model.

[0026] Machine learning models can be configured to determine the predicted location of each object in an environment (e.g., a physical area in which a vehicle operates and / or a simulated environment). In some examples, the environment can be determined based on sensor data from one or more sensors associated with the vehicle. The object locations predicted by the models described herein can be based on passive prediction (e.g., actions taken independently of the vehicle and / or another object in the environment, substantially no response to actions of the vehicle and / or other objects, etc.), active prediction (e.g., based on responses to actions of the vehicle and / or another object in the environment), or a combination thereof.

[0027] As described herein, a model can refer to a machine learning model, a statistical model, a heuristic model, or a combination thereof. That is, a model can refer to a machine learning model that learns from a training dataset to improve the accuracy of its output (e.g., predictions). Additionally or alternatively, a model can refer to a statistical model that represents a logical and / or mathematical function that generates approximate values ​​that can be used to make predictions.

[0028] The techniques discussed in this paper can improve the functionality of vehicle computing systems in a variety of ways. Vehicle computing systems can determine the actions an autonomous vehicle should take based on predicted location data associated with one or more objects. In some examples, using the location prediction techniques described in this paper, the model can output object locations and associated probabilities that improve vehicle safety by characterizing the future locations of objects in more detail and with greater accuracy compared to previous models. Furthermore, by clustering object locations relative to vehicle behavior or position, object locations that will cause different vehicle actions in the future can be considered during operation planning.

[0029] The techniques discussed in this paper can further improve the functionality of computing devices in a variety of additional ways. In some cases, representing the environment and objects as feature vectors for the purpose of generating predicted locations can represent a simplified representation of the environment. In other cases, evaluating the output of the model can allow autonomous vehicles to generate more accurate and / or safer trajectories for traversing the environment with less computational resources. In at least some of the examples described in this paper, predicted object locations can take into account object-to-object dependencies and / or relatively rare actions of objects, resulting in safer decisions for the system. These and other improvements to the functionality of computing devices are discussed in this paper.

[0030] The methods, apparatuses, and systems described herein can be implemented in a variety of ways. Example implementations are provided below with reference to the accompanying figures. Although some of the examples below are discussed in the context of autonomous vehicles, the methods, apparatuses, and systems described herein can be applied to a wide range of systems. In one example, a machine learning model can be utilized in a driver-controlled vehicle, where such a system can provide indications as to whether performing various maneuvers is safe. In another example, the methods, apparatuses, and systems can be used in an aviation or marine environment. Additionally or alternatively, the techniques described herein can be used with real-world data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0031] Figure 1 An autonomous vehicle (vehicle 102) in example environment 100 is illustrated, where an example machine learning model (prediction component 104) can process input data to predict object locations. As shown, vehicle 102 includes prediction component 104, which represents one or more models (e.g., machine learning models) for processing various types of input data 106 (e.g., vehicle information, object information, map data, etc.) associated with one or more objects in environment 100, and determines output data 108 representing object locations, vehicle locations, and / or clustering information that can be used for simulation, vehicle control, etc. In some examples, the prediction techniques described herein may be implemented at least in part by or associated with vehicle computing devices (e.g., vehicle computing device 504) and / or remote computing devices (e.g., computing device 534).

[0032] In various examples, the vehicle computing device associated with vehicle 102 can be configured to detect one or more objects (e.g., objects 110 and 112) in environment 100, such as via sensing components. In some examples, the vehicle computing device can detect objects based on sensor data received from one or more sensors. In some examples, sensors can include sensors mounted on vehicle 102, and include, but are not limited to, ultrasonic sensors, radar sensors, light detection and ranging (LiDAR) sensors, cameras, time-of-flight (ToF) sensors, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, etc.), Global Positioning Satellite (GPS) sensors, etc. In various examples, vehicle 102 can be configured to send and / or receive data from other autonomous vehicles and / or other sensors in the environment. The data can include sensor data, such as data about objects detected in environment 100.

[0033] In various examples, the vehicle computing device can receive sensor data and perform semantic classification of detected objects (e.g., determine object type), such as whether the object is a pedestrian like object 110, a vehicle, building, truck, motorcycle, moped, etc., like object 112. Objects can include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects (e.g., other vehicles, pedestrians, cyclists, etc.). In some examples, the classification can include another vehicle (e.g., car, pickup truck, semi-truck, tractor unit, bus, train, etc.), pedestrians, children, cyclists, skateboarders, horse riders, animals, etc. In various examples, the model can use the classification of objects to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, the potential trajectory of an object can be considered based on its characteristics (e.g., how the object might move in the environment).

[0034] Typically, prediction component 104 provides functionality for determining candidate locations 114 for object 110, candidate locations (e.g., candidate locations 116 and 118) for object 112, and / or candidate location 120 for vehicle 102. Candidate locations can be considered as the future locations of the corresponding objects (or vehicles) and can include regions corresponding to the size of each object. Environment 100 can include any number of objects, and each object can be associated with one or more candidate locations. For example, object 110 may potentially occupy a location in environment 100 other than candidate location 114 at a certain point in time, and may also occupy various locations over a period of time (e.g., assuming pedestrian movement).

[0035] In some examples, the prediction component 104 can generate output data 108 for different future times. For example, at a given time, the prediction component 104 can generate output data 108 for different future times (e.g., every 0.1 seconds over four seconds, or some other time interval or frequency). In various examples, the prediction component 104 can iteratively determine the output data 108 for each future time based at least in part on the output data 108 associated with previous times. In other words, the prediction component 104 can predict the location of an object for different future times and subsequently consider the object's potential actions at previous times.

[0036] The candidate position 120 of vehicle 102 can represent, for example, a predicted position that vehicle 102 will navigate to in the future (e.g., six seconds or other time range). In some examples, prediction component 104 can receive candidate position 120 as input data. Prediction component 104 can, for example, receive location data including one or more future vehicle positions. In various examples, prediction component 104 can receive vehicle trajectory 122 and determine positions along the vehicle trajectory at different times over a period of time, including candidate position 120. For example, vehicle 102 may occupy various positions before reaching candidate position 120, and prediction component 104 can identify candidate positions for vehicle 102 at different future times.

[0037] In various examples, candidate positions of one or more objects (e.g., candidate positions 114, 116, and 118) may be predicted, generated, or otherwise determined by prediction component 104 based at least in part on candidate positions of vehicle 102. For example, prediction component 104 may receive candidate position 120 as input data 106 (e.g., from prediction component, planning component, or other component) and output object candidate positions that, if occupied by an object, invoke different reactions of vehicle 102 to avoid the corresponding object.

[0038] As an example and not a limitation, prediction component 104 may determine object candidate locations at least in part based on vehicle candidate locations. For example, prediction component 104 may cluster object candidate locations relative to candidate location 120 such that the response of vehicle 102 is identical for each object candidate location in the corresponding cluster. Figure 1 In this context, considering that candidate position 114 does not affect the process of vehicle 102 reaching candidate position 120 (e.g., a first vehicle behavior), candidate position 114 of pedestrian object 110 can be in a different cluster than candidate position 118 of vehicle object 112, while candidate position 118 can cause vehicle 102 to brake (e.g., a second vehicle behavior). Further discussion of clustering can be found throughout this disclosure, including... Figure 2 See Figure 6.

[0039] In some examples, prediction component 104 may be configured to receive and / or determine vector representations of one or more of the following: environmental data (e.g., top-down view data), object state, and vehicle state. For example, prediction component 104 may include a machine learning model (e.g., a graph neural network (GNN)) to generate one or more vectors representing features of the environment (e.g., roads, crosswalks, buildings, etc.), the current state of an object (e.g., pedestrian object 110 and / or vehicle object 112), and / or the current state of vehicle 102. In other instances, feature vectors may represent rasterized images based on top-down view data. Additional details regarding the inputs to prediction component 104 are provided throughout this disclosure. Additional details regarding the use of GNNs to predict object locations are described in U.S. Patent Application Serial No. 17 / 535,357, filed November 24, 2021, entitled “Encoding Relative Object Information Into Node Edge Features,” which is incorporated herein by reference in its entirety and for all purposes.

[0040] The output data 108 from the prediction component 104 can be used by the vehicle computing device in various ways. For example, the prediction component 104 can transmit the output data 108 to the planning component 124 of the vehicle computing device to control the vehicle 102 in the environment 100 (e.g., determine candidate vehicle trajectories and / or control the propulsion, braking, or steering systems). In some examples, the planning component 124 can determine planning data for the vehicle 102 to navigate in the environment. The planning data may include one or more vehicle trajectories (candidate trajectories for avoiding objects), one or more object trajectories, to name just a few. The planning data may also, or alternatively, represent determinations made by a decision tree configured with reference actions corresponding to different object positions output from the prediction component 104.

[0041] Training components of remote computing devices, such as computing device 534 (not shown) and / or vehicle computing device 504 (not shown), can be implemented as training prediction component 104. Training data can include various data associated with values ​​(e.g., expected classifications, inferences, predictions, etc.), such as image data, video data, LiDAR data, radar data, audio data, other sensor data, etc. In some examples, training data can include determinations based on sensor data, such as bounding boxes (e.g., two-dimensional and / or three-dimensional bounding boxes associated with objects), segmentation information, classification information, object trajectories, etc. Such training data can generally be referred to as "ground truth." For example, training data can be used for image classification and therefore can include images of the environment captured by an autonomous vehicle and associated with one or more classifications. In some examples, such classification can be based on user input (e.g., user input indicating that the image depicts a particular type of object) or can be based on the output of another machine learning model. In some examples, such labeled classification (or more generally, the output of labeled values ​​associated with the training data) can be referred to as ground truth.

[0042] Figure 2 An example block diagram of an example computer architecture 200 for implementing the techniques for generating example feature vectors as described herein is shown. The example computer architecture 200 includes a computing device comprising... Figure 1 The prediction component 104 is used in the example; however, in other examples, the techniques described in example computer architecture 200 can be performed by vehicle computing device 504 and / or computing device 534. Feature vectors can be used, for example, as input to prediction component 104.

[0043] As shown, the computing device can receive and / or determine data associated with environment 202 (or alternatively, example environment 100). For example, the computing device can receive data about objects in the environment from sensing component 522, and can receive data about the environment itself from positioning component 520, sensing component 522, and one or more maps 528. By way of example and not limitation, the computing device can receive sensor data associated with autonomous vehicle 204 and object 206, and determine data including a top-down representation 208 of environment 202 and / or a vector representation 210 of environment 202 based at least in part on the sensor data.

[0044] In some examples, the vector representation 210 (e.g., feature vectors) can be determined by a graph neural network, a class of neural networks that operate on a graph structure. In various examples, the graph neural network can be partially or fully connected and has individual edge features associated with different pairs of nodes in the graph neural network. Machine learning-based inference operations can be performed to update the state of the graph neural network, including updating node and / or edge features based on internal inputs determined from the graph neural network itself and / or on updated observations perceived by the autonomous vehicle in the environment. Updates to the graph neural network can represent a predicted future state of the environment, and the autonomous vehicle can decode portions of the graph neural network to determine predictions of the predicted state of entities in the environment, including their positions, velocities, trajectories, and / or other updates.

[0045] In some examples, vector representations can be determined based on polylines (e.g., a set of line segments) representing one or more map elements. For example, a graph neural network can encode and aggregate polylines into a node data structure representing map elements. For instance, objects or features of the environment can be represented by polylines (e.g., a lane can be segmented into multiple smaller line segments, whose lengths, positions, orientation angles (e.g., yaw), and directions define the lane when aggregated). Similarly, a pedestrian crossing can be defined by four connected line segments, and a road edge or centerline can be multiple connected line segments.

[0046] In this example, each polyline may include one or more points and / or line segments that can be represented as vectors. For example, each line segment in a road, lane, or pedestrian crossing may be defined by location, length, orientation, direction, and / or other attributes. Attributes associated with a line segment may be stored in a vector data structure representing that line segment, and each line segment in a polyline associated with the same map element may be encoded and aggregated into a node structure. In addition to the attributes associated with the individual line segments of the polyline (e.g., location, length, and orientation), additional attributes may be associated with the map element itself (e.g., map element type, orientation, permissibility, etc.). Further details of graph neural networks are described in U.S. Patent Application Serial No. 17 / 187,170, which is incorporated herein by reference in its entirety.

[0047] Typically, the top-down representation 208 can represent the area surrounding the autonomous vehicle 204. In some examples, this area can be based at least in part on the area visible to the sensors (e.g., sensor range), receding horizon, area associated with an action (e.g., crossing an intersection), etc. In some examples, the top-down representation 208 can represent a 100-meter by 100-meter area around the autonomous vehicle 204, but any area is conceivable. In various examples, the top-down representation 208 can be determined at least in part based on map data and / or sensor data captured from or associated with the sensors of the autonomous vehicle 204 in the environment 202.

[0048] In various examples, the top-down representation 208 of environment 202 can represent a top-down view of the environment and can include one or more multi-channel images, such as a first channel 212, a second channel 214, and a third channel 216. The computing device can generate or determine multi-channel images to represent different attributes of environment 202 with different channel images. For example, an image with multiple channels, where each channel represents some information (semantic or other). In some examples, one of the channel images (e.g., the first channel 212, the second channel 214, or the third channel 216) can represent vehicle position, object position, environmental features, object velocity 218, object heading, object acceleration, object yaw, object attributes, pedestrian crossing permission (e.g., pedestrian crossing light or audio status), and traffic light permission (e.g., traffic light status), to name just a few. In this way, the top-down representation 208 can represent objects in the environment (e.g., represented by bounding boxes, as discussed herein), semantic information about the objects (e.g., classification type), movement information (e.g., velocity information, acceleration information, etc.), etc. Additional details regarding the use of top-down representation are described in U.S. Patent Application Serial No. 16 / 504,147, entitled “Prediction on Top-Down Scenes Based OnAction Data”, filed July 5, 2019.

[0049] Figure 2The environment 202 is shown to be represented, or alternatively, by a vector representation 210, which includes vectors representing features of the object and / or environment, including one or more of the following: attributes of the object 206 (e.g., position, velocity, acceleration, yaw, etc.), the history of the object 206 (e.g., position history, velocity history, etc.), attributes of the autonomous vehicle 204 (e.g., velocity, position, etc.), the history of the autonomous vehicle 204 (e.g., position history, velocity history, etc.), and / or features of the environment 202 (e.g., road boundaries, road centerlines, pedestrian crossing permission, traffic light permission, etc.). For example, vector representation 210 may include vectors representing features of the environment, including a road boundary vector 220 and a road centerline vector 222.

[0050] In various examples, example computer architecture 200 may include a computing device that generates vector representation 210 based at least in part on state data associated with autonomous vehicle 204 and / or object 206. The state data may include descriptions of objects (e.g., object 206 or...) in an environment (e.g., in example environment 100). Figure 1 Data for pedestrian objects 110, vehicle objects 112, and / or vehicles (e.g., vehicle 102) in the context. In various examples, state data may include one or more of the following associated with the object and / or vehicle: position data, orientation data, heading data, speed data, rate data, acceleration data, yaw rate data, or steering rate data. In some examples, vectors associated with the environment, vehicle state, and / or object state may be combined into a vector representation 210 (e.g., vectors may be concatenated).

[0051] In some examples, the top-down representation 208 can be fed into a machine learning model 224 (e.g., a convolutional neural network (CNN)), which determines a feature vector 226 for input into the prediction component 104. Additionally or alternatively, the vector representation 210 can be fed into a machine learning model (e.g., a graph neural network (GNN)) that determines a feature vector 230 for input into the prediction component 104. Feature vector 226 can represent the association between one or more channels of the top-down representation 208, while feature vector 230 can represent the association between vectors of the vector representation 210. The prediction component 104 can process feature vectors 226 and / or 230 to determine output data 108 indicating the predicted location of object 206 or another object different from object 206.

[0052] In various examples, the computing device generates feature vectors based at least in part on state data associated with vehicles and / or objects. State data may include descriptions of objects in the environment (e.g., in example environment 100), Figure 1 The data includes pedestrian object 110, vehicle object 112, and / or vehicle (e.g., vehicle 102). In various examples, the status data may include one or more of the following associated with the object and / or vehicle: position data, orientation data, heading data, speed data, rate data, acceleration data, yaw rate data, or steering rate data.

[0053] In some examples, sensor data or processed sensor data (e.g., top-down representation) can be fed into a machine learning model (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), a graph neural network (GNN), etc.) that can determine feature vectors for processing by the machine learning model.

[0054] In various examples, the top-down representation 208 and / or vector representation 210 may include the predicted location of the autonomous vehicle 204 (shown as location 232 in the top-down representation 208). The predicted location may represent a position that the autonomous vehicle 204 may occupy in the future and may be determined based on the historical location of the autonomous vehicle 204 (and / or another autonomous vehicle in the fleet) in a previous time period. In some examples, the predicted location may represent a location determined by the prediction and / or planning components of the vehicle computing device.

[0055] Figure 3 An example block diagram 300 of an example computer architecture is shown for implementing techniques for determining the location of example objects as described herein. For example, computing device 302 includes... Figure 1 The prediction component 104 further includes a clustering component 304 and an analysis component 306. In various examples, the computing device 302 may receive input data 308 for processing by one or more of the clustering component 304 and / or analysis component 306, and determine output data 310 representing one or more object locations that an object may occupy at a future time. In some examples, the computing device 302 may be associated with a vehicle computing device 504 and / or a computing device 534. In some examples, regarding... Figure 3 The described technology can be executed when vehicle 102 navigates in environment 100 (e.g., a real-world environment or a simulated environment).

[0056] In various examples, computing device 302 may receive input data 308 and predict object locations in the environment relative to the predicted location of vehicle 102. For example, prediction component 104 may receive historical data indicating locations previously occupied by one or more objects and map data representing the environment. Prediction component 104 may also or alternatively receive vehicle data indicating the previous and / or predicted locations of vehicles (e.g., autonomous vehicles). In some examples, prediction component 104 may determine a first distribution or dataset of object locations and a second distribution or dataset of vehicle locations associated with future time. Prediction component 104 may determine output data 310 based at least in part on comparing object locations in the first distribution with vehicle locations in the second distribution. In various examples, the probability of a target set of vehicles may be determined based on the historical set of all objects (including vehicles) and the target state associated with one of the objects. Mathematically, such a probability may be defined as p(T h | H a ….H h , t ka ), where the subscript "h" is relative to the vehicle, and "a" is relative to the object. Similarly, the target t of the object. ka It can be expressed as the probability p(t) ka | H a …H h In some examples, given the history of an object and / or vehicle, the probability of a target object can be constrained based on kinematically feasible transformations. In any case, both models can be machine learning models (e.g., neural networks or others) and trained based on observed data.

[0057] In various examples, computing device 302 can implement clustering component 304 to determine the location of one or more objects in an environment (e.g., p(t)). ka (output of )). For example, clustering component 304 can represent a model (e.g., based on historical data, map data, vehicle data, etc.) for clustering object locations predicted by prediction component 104.

[0058] In various examples, clusters can be determined based on the predicted location of the vehicle. Once clusters are determined, scores can be assigned to one or more individual clusters within that set of clusters. As a non-limiting example, a score can include the sum of all probabilities at all locations within a cluster. Once scored, a location can be selected from each cluster. In some examples, such a location can be selected based on, for example, a cluster center, the most likely target of the cluster, a random target, etc. The location selected in this way can then be used to determine the possible location of the vehicle, as described above. For example, clustering component 304 can determine a first cluster of at least some candidate locations from a set of candidate locations associated with the object, the candidate locations of which in the first cluster cause a first behavior of the autonomous vehicle in the future, and determine a second cluster of at least some candidate locations from a set of candidate locations associated with the object, the candidate locations of which in the second cluster cause a second vehicle behavior of the autonomous vehicle in the future. The first vehicle behavior and / or the second vehicle behavior can represent braking, steering, acceleration, or other actions of the autonomous vehicle.

[0059] As an example, and not a limitation, clustering component 304 can determine various behaviors or actions that a vehicle might need to take to reach its predicted location. For example, a vehicle might control its braking system, change lanes, or remain stationary to navigate to the predicted location, taking into account traffic regulations, road boundaries, and other features obtained using map data. In some examples, the locations of objects that might influence the vehicle's future actions can be considered independently of where the objects were located before reaching their respective locations.

[0060] In some examples, the clusters output by clustering component 304 can be based on the vehicle performing a specific action in response to an object occupying one of the candidate positions in the corresponding cluster. Clustering component 304 can assign, select, or otherwise determine candidate object positions to be included in a first cluster or a second cluster based on the autonomous vehicle's response to an object being in one of the candidate positions in the corresponding cluster (e.g., the vehicle can brake in response to an object in the first cluster, or change lanes in response to an object in the second cluster).

[0061] For example, clustering component 304 can determine the clustering of object locations relative to predicted vehicle locations based on the fact that each object location in the corresponding cluster causes the same vehicle action. For instance, the model can assign multiple object locations to a cluster based on the fact that vehicle actions at future times are the same for each candidate object location in the cluster. In some examples, to save computational resources, object locations can be determined independently of knowing the object's starting location or tracking the object to that candidate object location. Therefore, predicted vehicle locations can be used to regulate how object locations are clustered.

[0062] In various examples, computing device 302 may implement analysis component 306 to analyze input data 308 and / or output from clustering component 304 (e.g., object location data associated with one or more clusters). For example, analysis component 306 may determine the difference between the distribution of object locations and vehicle locations, and compare the difference to a difference threshold to identify a subset of object locations most relevant to vehicles. In various examples, analysis component 306 may reduce the number of predicted object locations based at least in part on this comparison.

[0063] In some examples, analysis component 306 may determine the probability that an object occupies a specific candidate object location, for example, based on historical object locations in similar or identical areas of the environment. Analysis component 306 may also, or alternatively, determine the probability that an object occupies a specific cluster to determine a subset of candidate object locations as output data 310. Analysis component 306 may use probabilities to reduce the number of candidate object locations as output data 310 (e.g., a threshold number of object locations may be selected, where the first M candidate object locations with a probability range are taken, where M is an integer).

[0064] like Figure 3 As depicted, input data 308 may represent one or more of the following: map data, vehicle data, sensor data, classification data, prediction data, planner data, and / or environmental data, but other data may also be considered. In some examples, input data 308 may include predicted vehicle location information of vehicle 102, and clustering component 304 may determine clusters of object locations based on predicted vehicle location information (e.g., one or more future locations of vehicle 102).

[0065] In some examples, multiple vector representations (e.g., feature vector 226 and / or feature vector 230) may be input into prediction component 104 to determine output data 310 indicating the predicted location of multiple objects in the environment.

[0066] Output data 310 may represent one or more of the following: object location (e.g., predicted object location, candidate object location), vehicle location (e.g., predicted vehicle location, distribution of vehicle locations, etc.), and / or clustering information (e.g., data describing clusters associated with object locations). In some examples, output data 310 may be associated with clusters, datasets, distributions, etc.

[0067] The data output by computing device 302 can be used in various ways. For example, output data representing object location can be used to perform simulations, control vehicles, and / or verify or test vehicle performance, to name just a few. For example, output data 310 can be used, at least in part, to verify the outputs of components configured to control a vehicle in a future time environment, based on simulation results. For example, simulation results can be used to verify determinations or predictions from planning components, prediction components, etc. In some examples, at least a portion of output data 310 can be stored in a storage device for later access. For example, information about object location and / or clustering can be stored in a database to be used as input data and / or training data. Object location can, for example, represent the location of an object in an environment that it may occupy in the future.

[0068] In some instances, computing device 302 may transmit at least a portion of output data 310 to a computing device or component configured to control an autonomous vehicle. For example, object location may be transmitted to a planning component of a vehicle computing device configured to determine planning data (e.g., vehicle trajectory, object trajectory, decision tree output, etc.).

[0069] In some examples, output data 310 can be used to verify or test the performance of the vehicle. For example, a safety system such as a collision avoidance system can receive object position data for processing. Additionally or alternatively, the performance of a vehicle controller configured to control some aspects of the vehicle (e.g., braking system, acceleration system, etc.) can be verified by having the vehicle controller receive output data 310 as input.

[0070] The object locations associated with the output data 310 can be configured for use in the decision tree to control the vehicle. For example, one or more object locations can be associated with corresponding reference actions or nodes in the decision tree. As mentioned, different object locations can be associated with different vehicle actions, so including object locations in the decision tree as reference actions can improve vehicle planning operations by implementing more realistic reference actions in the decision tree (e.g., considering object locations associated with different vehicle behaviors to plan for greater variations in potential object actions).

[0071] Figure 4A shows an example block diagram 400 of a computing device that implements an example prediction component for generating object location information. For example, computing device 302 includes... Figure 1 The prediction component 104 further includes a clustering component 304 and an analysis component 306 to determine output data 108 representing the location of the predicted object. In some examples, the computing device 302 may be associated with the vehicle computing device 504 and / or the computing device 534.

[0072] Figure 4AVehicle 102 is also shown in association with vehicle trajectory 402, vehicle position 404, and vehicle position 406. Vehicle trajectory 402 may represent a predicted trajectory from a vehicle computing device configured to control vehicle 102. In various examples, vehicle trajectory 402 may include attitude or heading, acceleration, and / or velocity associated with vehicle 102. Vehicle position 404 and vehicle position 406 may represent the predicted position of vehicle 102 at some future time. In some examples, prediction component 104 may receive vehicle trajectory 402, vehicle position 404, and / or vehicle position 406 as input data.

[0073] In various examples, computing device 302 may receive map data representing the environment, including objects 408 and 410 (e.g., additional vehicles). In various examples, objects 408 and 410 may be detected, at least in part, based on sensor data from one or more sensors associated with vehicle 102 (e.g., during previous navigation in the environment). Additionally or alternatively, computing device 302 may receive historical data representing the positions of objects in the environment over a previous time or period. Such historical positions of all objects (which may include vehicles) may be fed into a machine learning model to determine a set of potential targets (e.g., future states—position, speed, orientation, etc.) associated with the objects. In various examples, clustering component 304 may determine a first cluster 412, a second cluster 414, and / or a third cluster 416 (collectively, “clusters”) of this set of potential targets for the vehicle. Figure 4A A first cluster 412 is shown, comprising positions 418 (e.g., 418(1), 418(2), ..., 418(N), where N is an integer). For example, positions 418 in the first cluster 412 may represent the distribution of predicted (or candidate) object locations that cause the same or similar vehicle behavior. For clarity and brevity, these positions are shown relative to the first cluster 412, but each corresponding cluster may include multiple object locations.

[0074] In some examples, clustering component 304 may associate, select, or otherwise determine object locations to be included in a cluster based on one or more future locations of vehicle 102 (e.g., vehicle trajectory 402, vehicle position 404, and / or vehicle position 406). In various examples, clusters may include different numbers of object locations and may vary in size. As an example, clustering component 304 may select at least some object locations from a distribution of object locations output by the model based on whether an object location (e.g., position 418) would cause the vehicle to take a first action (e.g., braking to avoid the object) if it were occupied by an object in the future. In some examples, object 408 may be as follows: Figure 4AThe dashed lines indicate a U-turn, and vehicle 102 can plan for the probability that object 408 can reach a position within the first cluster 412. In other examples, object 408 can go straight at the intersection to occupy a position in the second cluster 414 or the third cluster 416, or turn left to occupy the third cluster 416. It can be seen that the third cluster 416 includes predictions of object 408 continuing straight and turning left. These predictions are clustered because both require vehicle 102 to yield to object 408 (e.g., by waiting and / or stopping at the intersection before continuing).

[0075] In various examples, output data 310 (e.g., a subset of object location distributions) may include locations from multiple clusters (e.g., two or more of a first cluster 412, a second cluster 414, or a third cluster 416). For example, analysis component 306 may select the most probable location from the available locations in each corresponding cluster. In various examples, analysis component 306 may determine the probability that an object occupies a location and compare these probabilities with each other to select a location.

[0076] In some examples, the analysis component 306 may also, or alternatively at least in part, determine the threshold number of object locations to be included as output data 310 based on one or more criteria. Criteria may include, for example, determining the minimum number of clusters, the maximum number of clusters, the number of locations from different clusters, etc., for planning purposes for different vehicle actions. Criteria may also, or alternatively, include vehicle state data, traffic control information (e.g., right-of-way, etc.), distance information (e.g., between objects and / or vehicles), time information (e.g., the time when objects and / or vehicles arrive at their locations), etc. In some examples, the analysis component 306 may determine the threshold number of object locations to be included as output data 310 based on criteria representing one or more costs. Costs may be associated with vehicle behavior (e.g., to ensure representation of a minimum number of vehicle behaviors) and / or clustering (e.g., the cost of omitting or including clusters), to name just a few.

[0077] By identifying clusters and processing subsets of locations from those clusters, the output data 310 can represent multiple different actions an object can take, regardless of whether the object is a pedestrian or another vehicle, and regardless of probability. Therefore, a vehicle can consider all potential locations of an object when deciding how to navigate. This consideration provides improvements, for example, allowing the evaluation of locations associated with low-probability (but still likely) clusters despite limited computational resources.

[0078] In some examples, prediction component 104 may receive or generate a vehicle location distribution representing different possible locations of vehicle 102 at some future time. The locations of the vehicle distributions may be compared to each other to determine modified vehicle locations (e.g., average location, mean location, or other relationships between vehicle locations). In some examples, the modified vehicle locations may be used as vehicle location 404. Additionally or alternatively, clustering component 304 may cluster a threshold number of candidate object locations relative to the modified candidate vehicle locations to determine a first cluster 412, a second cluster 414, etc.

[0079] In some examples, the analysis component 306 may determine a subset of object locations from candidate object locations (whether part of a clustered or non-clustered distribution) based on a first probability that an object occupies a first object location 418(1) in the first cluster 412 and a second probability that an object occupies a second object location 418(2) in the first cluster 412. In some examples, the analysis component 306 may select either a first (candidate) object location or a second (candidate) object location as a subset of object locations from the first cluster 412 based at least in part on the first and second probabilities.

[0080] In various examples, analysis component 306 may determine a first score for an object (e.g., object 408 or object 410) to occupy a first cluster 410 in the future and a second score for an object to occupy a second cluster 412 in the future, and so on for each determined cluster. In such examples, analysis component 306 may determine a subset of object locations based at least in part on the first and second scores. In some examples, output data 310 may include a set of clusters with various scores to capture low-probability actions of objects. In some examples, scores may represent the similarity between candidate locations of objects in a cluster and vehicle locations (e.g., distance from vehicle location 404).

[0081] Despite Figure 4A The vehicle location is depicted as adjacent to the cluster, but in various examples, vehicle location 404 and / or vehicle location 406 may be included in the cluster (e.g., within the cluster boundary).

[0082] In some examples, cluster scores can represent the relevance of locations within a cluster to vehicles, and can be based on factors such as the probability of an object occupying a corresponding location, the cost of including or excluding a cluster as output data (to name just a few), object state data of objects associated with the cluster, vehicle state data, right-of-way information associated with (multiple) objects in the cluster, etc. For example, analysis component 306 can determine the probabilities of various object locations and further determine scores based on the mean, average, and / or maximum probability of each cluster, or other calculations. In some examples, analysis component 306 can select object locations from one or more clusters as output data 310 (e.g., based on scores, probabilities, etc.). As an example, and not a limitation, object locations can be selected from clusters to represent reference actions in tree search.

[0083] In some examples, cluster scores can indicate whether locations from a cluster should be included in a subset of the object locations represented by output data 310. For example, analysis component 306 can compare scores to each other to determine whether to select one or more locations from the corresponding clusters. Analysis component 306 can, for example, determine whether to include locations from clusters associated with relatively low scores (e.g., scores lower than the score of another cluster) to ensure that output data 310 includes locations associated with less likely vehicle behavior.

[0084] Figure 4B Example block diagram 400 of a computing device implementing an example clustering component for generating object location information is shown. In various examples, analysis component 306 may determine a dataset 420 including locations 422, where locations 422 include locations 422(1), 422(2), ... up to 422(N), where N is an integer. In some examples, locations 422 may correspond to the locations of corresponding objects visualized in three-dimensional space for easier interpretation; however, it should be understood that the parameter space used for clustering may contain any number of parameters and is not limited in size or dimension. In various examples, analysis component 306 may generate locations 422 to represent different candidate locations of objects in the environment and compare locations 422 with each other in various ways to determine differences and / or similarities. In various examples, the number of locations in dataset 420 may vary so that various different attributes are available for comparison. For example, clustering component 304 may perform clustering (e.g., implement a clustering algorithm) to associate locations 422 with clusters 424 that may include any number of locations. Although a single cluster is shown, clustering component 304 can identify or determine any number of clusters.

[0085] In various examples, analysis component 306 can compare the attributes of the locations of clusters 424 with each other and / or with vehicle locations 404 to identify the most likely location of objects (e.g., average, mean, or other calculations). By comparing locations 422 with each other and / or with vehicle locations 404, a reduced dataset is processed that still represents the locations within clusters 424.

[0086] In some examples, the analysis component 306 can determine the distance 426 between location 428(1) at the center of cluster 424 and location 428(2) (and optionally, other points within the boundaries of cluster 424). The distance 426 can represent the similarity between a first attribute associated with location 428(1) and a second attribute associated with location 428(2) (e.g., points closer to location 428(1) may be more similar than points farther away from location 428(1)).

[0087] Although cluster 424 is shown as a sphere in the example, other shapes and / or sizes are possible depending on the number of parameters in the parameter space (e.g., dataset 420). Typically, cluster 424 can represent a subset of points identified as having a relationship based on clustering techniques (e.g., causing the same vehicle motion).

[0088] Figure 5 This is a block diagram of an example system 500 for implementing the techniques described herein. In at least one example, system 500 may include a vehicle, such as vehicle 502.

[0089] Vehicle 502 may include vehicle computing device 504, one or more sensor systems 506, one or more transmitters 508, one or more communication connections 510, at least one direct connection 512, and one or more drive systems 514.

[0090] Vehicle computing device 504 may include one or more processors 516 and memory 518 communicatively coupled to one or more processors 516. In the illustrated example, vehicle 502 is an autonomous vehicle; however, vehicle 502 may be any other type of vehicle, such as a semi-autonomous vehicle or any other system with at least image capture equipment (e.g., a camera-enabled smartphone). In some instances, autonomous vehicle 502 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the journey and where driver (or occupant) control of the vehicle is not anticipated at any time. However, in other examples, autonomous vehicle 502 may be a fully or partially autonomous vehicle with any other level or classification.

[0091] In various examples, vehicle computing device 504 may store sensor data associated with the actual position of an object at the end of a set of estimated states (e.g., the end of a time period), and may use this data as training data to train one or more models. In some examples, vehicle computing device 504 may provide the data to a remote computing device (i.e., a computing device separate from the vehicle computing device, such as computing device 534) for data analysis. In such examples, the remote computing device may analyze the sensor data to determine the actual position, velocity, direction of travel, etc., of the object at the end of the set of estimated states. Additional details on training machine learning models based on stored sensor data by minimizing the difference between the actual position and the predicted position and / or predicted trajectory are described in U.S. Patent Application Serial No. 16 / 282,201, filed March 12, 2019, entitled “Motion Prediction Based on Appearance,” the entire contents of which are incorporated herein by reference.

[0092] In the example shown, the memory 518 of the vehicle computing device 504 stores a positioning component 520, a perception component 522, a planning component 524, one or more system controllers 526, one or more maps 528, and a model component 530. The model component 530 includes one or more models, such as a first model 532A, a second model 532B, up to an Nth model 532N (collectively referred to as "model 532"), where N is an integer. Although for illustrative purposes... Figure 5 The system is depicted residing in memory 518, but it is contemplated that the positioning component 520, perception component 522, planning component 524, one or more system controllers 526, one or more maps 528, and / or model component 530 including model 532 may additionally or alternatively be accessible by vehicle 502 (e.g., stored in memory remote from vehicle 502 or otherwise accessible by memory remote from vehicle 502, such as, for example, in memory 538 of remote computing device 534). In some examples, model 532 may provide functionality associated with prediction component 104. In some examples, model 532 may include one or more of the following: machine learning model, statistical model, heuristic model, or a combination thereof.

[0093] In at least one example, the localization component 520 may include the ability to receive data from the sensor system 506 to determine the position and / or orientation (e.g., one or more of x, y, z position, roll, pitch, or yaw) of the vehicle 502. For example, the localization component 520 may include and / or request / receive a map of the environment, such as from map 528 and / or map component 544, and may continuously determine the position and / or orientation of the autonomous vehicle within the map. In some cases, the localization component 520 may utilize SLAM (Simultaneous Localization and Mapping), CLAMS (Simultaneous Calibration, Localization and Mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position of the autonomous vehicle. In some cases, the localization component 520 may provide data to various components of the vehicle 502 to determine the initial position of the autonomous vehicle for determining the relevance of objects to the vehicle 502, as discussed herein.

[0094] In some cases, the perception component 522 may include functions for performing object detection, segmentation, and / or classification. In some examples, the perception component 522 may provide processed sensor data indicating the presence of objects (e.g., entities) near the vehicle 502 and / or the classification of objects as object types (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In some examples, the perception component 522 may provide processed sensor data indicating the presence of stationary entities near the vehicle 502 and / or the classification of stationary entities as a type (e.g., building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 522 may provide processed sensor data indicating one or more features associated with detected objects (e.g., tracked objects) and / or the environment in which the objects are located. In some examples, features associated with an object may include, but are not limited to, x-position (global and / or local), y-position (global and / or local), z-position (global and / or local), orientation (e.g., roll, pitch, yaw), object type (e.g., classification), object velocity, object acceleration, object range (size), etc. Features associated with the environment may include, but are not limited to, the presence of another object in the environment, the state of another object in the environment, time of day, day of week, season, weather conditions, dark / light indicators, etc.

[0095] Typically, the planning component 524 can determine the path that vehicle 502 will follow to traverse the environment. For example, the planning component 524 can determine various routes and trajectories, as well as various levels of detail. For example, the planning component 524 can determine a route from a first location (e.g., the current location) to a second location (e.g., the target location). For the purposes of this discussion, the route may include a sequence of waypoints for traveling between the two locations. As a non-limiting example, waypoints include streets, intersections, Global Positioning System (GPS) coordinates, etc. Furthermore, the planning component 524 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 524 can determine how to guide the autonomous vehicle from a first waypoint in the waypoint sequence to a second waypoint in the waypoint sequence. In some examples, the instructions may be a trajectory or a portion of a trajectory. In some examples, multiple trajectories may be generated substantially simultaneously (e.g., within technical tolerances) based on back-of-view techniques, with one of the multiple trajectories selected for navigation of vehicle 502.

[0096] In some examples, the planning component 524 may include a prediction component to generate predicted trajectories of objects in the environment (e.g., objects) and / or generate predicted candidate trajectories for vehicle 502. For example, the prediction component may generate one or more predicted trajectories for objects within a threshold distance of vehicle 502. In some examples, the prediction component may measure the trajectory of an object and generate the object's trajectory based on observed and predicted behavior.

[0097] In at least one example, the vehicle computing device 504 may include one or more system controllers 526, which may be configured to control the steering, propulsion, braking, safety, transmitter, communication, and other systems of the vehicle 502. The system controllers 526 may communicate with and / or control the corresponding systems of the drive system 514 and / or other components of the vehicle 502.

[0098] The memory 518 may also include one or more maps 528 that can be used by the vehicle 502 to navigate within the environment. For the purposes of this discussion, the map can be any number of data structures modeled in two, three, or N dimensions, capable of providing information about the environment, such as, but not limited to, topology (e.g., intersections), streets, mountains, roads, terrain, and general environment. In some instances, the map may include, but is not limited to: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information) etc.), intensity information (e.g., LiDAR information, radar information, etc.); spatial information (e.g., image data projected onto a grid, individual "surfels" (e.g., polygons associated with individual color and / or intensity)), reflectivity information (e.g., specular reflection information, retroreflection information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the vehicle 502 may be controlled at least in part based on the map 528. In other words, map 528 can be used in conjunction with positioning component 520, perception component 522 and / or planning component 524 to determine the position of vehicle 502, detect objects in the environment, generate routes, and determine actions and / or trajectories for navigation within the environment.

[0099] In some examples, one or more maps 528 may be stored on a remote computing device (e.g., computing device 534) accessible via network 540. In some examples, multiple maps 528 may be stored, for example, based on characteristics such as the type of entity, time of day, day of week, season of year, etc. Storing multiple maps 528 may have similar memory requirements but increases the speed at which data can be accessed in the maps.

[0100] like Figure 5 As shown, the vehicle computing device 504 may include a model component 530. The model component 530 may be configured to perform the functions of the prediction component 104, including predicting object locations, vehicle locations, clustering object locations relative to the predicted vehicle locations, etc. In various examples, the model component 530 may receive one or more features associated with detected objects from the perception component 522 and / or the sensor system 506. In some examples, the model component 530 may receive environmental characteristics (e.g., environmental factors, etc.) and / or weather characteristics (e.g., weather factors, such as snow, rain, ice, etc.) from the perception component 522 and / or the sensor system 506. Although in Figure 5 It is shown separately, but model component 530 may be part of planning component 524 or other components of vehicle 502.

[0101] In various examples, model component 530 may send predictions from one or more models 532, which may be used by planning component 524 to generate one or more predicted trajectories (e.g., direction of travel, speed, etc.) and / or one or more predicted trajectories (e.g., direction of travel, speed, etc.) of an object, such as from its prediction component. In some examples, planning component 524 may determine one or more actions (e.g., reference actions and / or sub-actions) of vehicle 502, such as vehicle candidate trajectories. In some examples, model component 530 may be configured to determine whether an object occupies a future position based at least in part on one or more actions of vehicle 502. In some examples, model component 530 may be configured to determine actions appropriate to the environment, such as based on environmental characteristics, weather characteristics, another object, etc.

[0102] Model component 530 can generate a set of estimated states for the vehicle and one or more detected objects over a period of time, extrapolated forward within the environment. Model component 530 can generate the set of estimated states for each action (e.g., a reference action and / or sub-action) determined to be applicable to the environment. The set of estimated states can include one or more estimated states, each including the estimated position of the vehicle and the estimated position of the detected objects. In some examples, the estimated states can include the estimated position of the detected objects at an initial time (T=0) (e.g., the current time).

[0103] The estimated location can be determined based on detected trajectories and / or predicted trajectories associated with the object. In some examples, the estimated location can be determined based on the assumption of a substantially constant velocity and / or a substantially constant trajectory (e.g., the object has little or no lateral movement). In some examples, the estimated location (and / or potential trajectory) can be based on passive and / or active predictions. In some examples, model component 530 can utilize physics- and / or geometry-based techniques, machine learning, linear-time logic, tree search methods, heatmaps, and / or other techniques for determining the predicted trajectory and / or estimated location of the object.

[0104] In various examples, estimated states can be generated periodically over a time period. For instance, model component 530 can generate estimated states at 0.1-second intervals over a time period. In another example, model component 530 can generate estimated states at 0.05-second intervals. The estimated states can be used by planning component 524 to determine the actions that vehicle 502 should take in the environment.

[0105] In various examples, model component 530 can utilize machine learning techniques to predict object positions, vehicle positions, etc. In such examples, a machine learning algorithm can be trained to determine, based on sensor data and / or previous predictions from the model, that an object is likely to behave in a particular manner relative to vehicle 502 at a specific time during a set of estimated states (e.g., a time period). In such examples, one or more of the vehicle 502 states (position, velocity, acceleration, trajectory, etc.) and / or object states, classifications, etc., can be input into such a machine learning model, and thus a trajectory prediction can be output by the model.

[0106] In various examples, model component 530 can use properties associated with each object type to determine the position, trajectory, velocity, or acceleration associated with the object. Examples of object type properties may include, but are not limited to: maximum longitudinal acceleration, maximum lateral acceleration, maximum vertical acceleration, maximum velocity, maximum directional change at a given velocity, etc.

[0107] It is understood that, for illustrative purposes, the components discussed herein (e.g., localization component 520, perception component 522, planning component 524, one or more system controllers 526, one or more maps 528, and model component 530 including model 532) are described separately. However, the operations performed by the various components may be combined or performed in any other component.

[0108] While examples are given where the techniques described herein are implemented by planning and / or modeling components of a vehicle, in some examples, some or all of the techniques described herein may be implemented by another system of the vehicle, such as an auxiliary safety system. Typically, such an architecture may include a first computing device for controlling vehicle 502 and an auxiliary safety system running on vehicle 502 to verify the operation of the main system and control vehicle 502 to avoid collisions.

[0109] In some cases, aspects of some or all of the components discussed herein may include any model, technique, and / or machine learning technique. For example, in some instances, components in memory 518 (and memory 538 discussed below) may be implemented as neural networks.

[0110] As described herein, an exemplary neural network is a technique for passing input data through a series of connected layers to produce an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As will be understood in the context of this disclosure, neural networks can utilize machine learning, which can refer to a broad category of such techniques in which outputs are generated based on learned parameters.

[0111] Although discussed in the context of neural networks, any type of machine learning consistent with this disclosure may be used. For example, machine learning techniques may include, but are not limited to, regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS), instance-based techniques (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angular regression (LARS), decision tree techniques (e.g., classification and regression trees (CART), iterative bisection method 3 (ID3), chi-square automatic interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian techniques (e.g., Naive Bayes, Gaussian Naive Bayes, multinomial Naive Bayes, average single dependency estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering techniques (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning techniques (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)). Deep learning techniques (e.g., Deep Boltzmann Machines (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Autoencoders), dimensionality reduction techniques (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projective Tracking, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA), Ensemble techniques (e.g., Boosting, Bagging, AdaBoost, Stacked Generalization, Gradient Boosting Machines (GBM), Gradient Boosting Regression Trees (GBRT), Random Forests, SVMs (Support Vector Machines), supervised learning, unsupervised learning, semi-supervised learning, etc.). Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, etc.

[0112] In at least one example, sensor system 506 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time-of-flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. Sensor system 506 may include multiple instances of each of these or other types of sensors. For example, lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of vehicle 502. As another example, camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of vehicle 502. Sensor system 506 may provide input to vehicle computing device 504. Additionally or alternatively, sensor system 506 may transmit sensor data to one or more computing devices 534 at a specific frequency via one or more networks 540 after a predetermined time period, in near real-time, etc.

[0113] Vehicle 502 may also include one or more transmitters 508 for emitting light and / or sound. Transmitters 508 may include internal audio and visual transmitters for communicating with the occupants of vehicle 502. By way of example, and not limitation, internal transmitters may include speakers, lights, signs, displays, touchscreens, haptic transmitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), etc. Transmitters 508 may also include external transmitters. By way of example, and not limitation, external transmitters may include lights (e.g., indicator lights, signs, light arrays, etc.) for signaling other indicators of direction of travel or vehicle movement, and one or more audio transmitters (e.g., speakers, speaker arrays, horns, etc.) for audible communication with pedestrians or other nearby vehicles, wherein one or more audio transmitters include acoustic beam steering technology.

[0114] Vehicle 502 may also include one or more communication connections 510 that enable communication between vehicle 502 and one or more other local or remote computing devices. For example, communication connection 510 may facilitate communication with other local computing devices and / or drive system 514 on vehicle 502. Furthermore, communication connection 510 may allow vehicle to communicate with other nearby computing devices (e.g., remote computing device 534, other nearby vehicles, etc.) and / or one or more remote sensor systems 542 for receiving sensor data. Communication connection 510 also enables vehicle 502 to communicate with remotely operated computing devices or other remote services at a distance.

[0115] Communication connection 510 may include physical and / or logical interfaces for connecting vehicle computing device 504 to another computing device or network (e.g., network 540). For example, communication connection 510 may enable Wi-Fi-based communication, such as via frequencies defined by the IEEE 502.11 standard, short-range wireless frequencies (such as Bluetooth), cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables the respective computing device to interface with other computing devices.

[0116] In at least one example, vehicle 502 may include one or more drive systems 514. In some examples, vehicle 502 may have a single drive system 514. In at least one example, if vehicle 502 has multiple drive systems 514, the individual drive systems 514 may be positioned at opposite ends of vehicle 502 (e.g., front and rear ends, etc.). In at least one example, drive system 514 may include one or more sensor systems to detect the condition of drive system 514 and / or the surrounding environment of vehicle 502. By way of example and not limitation, sensor systems may include: one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects in the surrounding environment of the drive module. Some sensors (e.g., wheel encoders) may be unique to drive system 514. In some cases, the sensor system on drive system 514 may overlap with or complement the corresponding system of vehicle 502 (e.g., sensor system 506).

[0117] The drive system 514 may include a number of vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate traction loss and maintain control, an HVAC system, lighting (e.g., illumination, such as headlights / taillights for illuminating the exterior of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high-voltage connectors, high-voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 514 may include a drive module controller that can receive and preprocess data from sensor systems and control the operation of various vehicle systems. In some examples, the drive module controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules to perform various functions of the drive system 514. In addition, the drive system 514 may include one or more communication connections that enable the respective drive module to communicate with one or more other local or remote computing devices.

[0118] In at least one example, the direct connection 512 can provide a physical interface for coupling one or more drive systems 514 to the body of the vehicle 502. For example, the direct connection 512 can allow the transfer of energy, fluid, air, data, etc., between the drive system 514 and the vehicle. In some cases, the direct connection 512 can also releasably secure the drive system 514 to the body of the vehicle 502.

[0119] In at least one example, the positioning component 520, sensing component 522, planning component 524, one or more system controllers 526, one or more maps 528, and model component 530 can process sensor data as described above and transmit their respective outputs to a computing device 534 via one or more networks 540. In at least one example, the positioning component 520, sensing component 522, planning component 524, one or more system controllers 526, one or more maps 528, and model component 530 can transmit their respective outputs to a remote computing device 534 at a specific frequency, after a predetermined time period, or in near real-time.

[0120] In some examples, vehicle 502 may transmit sensor data to computing device 534 via network 540. In some examples, vehicle 502 may receive sensor data from computing device 534 and / or remote sensor system 542 via network 540. Sensor data may include raw sensor data and / or processed sensor data and / or representations of sensor data. In some examples, sensor data (raw or processed) may be sent and / or received as one or more log files.

[0121] Computing device 534 may include a processor 536 and a memory 538 storing a map component 544, a sensor data processing component 546, and a training component 548. In some examples, map component 544 may include functionality to generate maps at various resolutions. In such examples, map component 544 may send one or more maps to vehicle computing device 504 for navigation purposes. In various examples, sensor data processing component 546 may be configured to receive data from one or more remote sensors (e.g., sensor system 506 and / or remote sensor system 542). In some examples, sensor data processing component 546 may be configured to process data and send the processed sensor data to vehicle computing device 504, for example, for use by model component 530 (e.g., model 532). In some examples, sensor data processing component 546 may be configured to send raw sensor data to vehicle computing device 504.

[0122] In some instances, training component 548 (e.g., according to...) Figure 4A and Figure 4B The techniques discussed herein (training) may include training a machine learning model to output the probability that an occluded region is free of any object, or that the occluded region is occupied by a static obstacle or a dynamic object. For example, training component 548 may receive sensor data representing an object passing through the environment over a period of time (e.g., 0.1 milliseconds, 1 second, 3 seconds, 5 seconds, 7 seconds, etc.). At least a portion of the sensor data may be used as input to train the machine learning model.

[0123] In some cases, training component 548 may be executed by processor 536 to train a machine learning model based on training data. Training data may include a variety of data associated with values ​​(e.g., expected classifications, inferences, predictions, etc.), such as sensor data, audio data, image data, map data, inertial data, vehicle state data, historical data (log data), or combinations thereof. Such values ​​may generally be referred to as “true values.” For example, training data may be used to determine the risk associated with occluded areas and may therefore include data representing the environment captured by the autonomous vehicle and associated with one or more classifications or determinations. In some examples, such classification may be based on user input (e.g., user input instructing the data to depict a specific risk) or may be based on the output of another machine learning model. In some examples, such a classification of labels (or more generally, the output of labels associated with the training data) may be referred to as a true value.

[0124] In some instances, training component 548 may include functionality to train a machine learning model to output classification values. For example, training component 548 may receive data representing labeled collision data (e.g., publicly available data, sensor data, and / or combinations thereof). At least a portion of the data may be used as input to train the machine learning model. Thus, by providing data on the vehicle traversing the environment, training component 548 can be trained to output occlusion values ​​associated with objects and / or occlusion regions, as discussed herein.

[0125] In some examples, training component 548 may include training data already generated by the simulator. For example, simulation training data may represent examples of a vehicle colliding with or narrowly missing objects in the environment to provide additional training examples.

[0126] The processor 516 of vehicle 502 and the processor 536 of computing device 534 can be any suitable processor capable of executing instructions to process data and perform the operations described herein. By way of example and not limitation, processors 516 and 536 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data to convert that electronic data into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors, provided they are configured to implement coded instructions.

[0127] Memory 518 and memory 538 are examples of non-transitory computer-readable media. Memory 518 and memory 538 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods described herein and the functions belonging to various systems. In various embodiments, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programming, and physical components, those shown in the accompanying figures being merely examples relevant to the discussion herein.

[0128] It should be noted that, although Figure 5 While shown as a distributed system, in an alternative example, components of vehicle 502 may be associated with computing device 534 and / or components of computing device 534 may be associated with vehicle 502. That is, vehicle 502 may perform one or more functions associated with computing device 534, and vice versa.

[0129] Figure 6A This is the first part of a flowchart depicting an example process 600 for determining the location of an object using one or more example models. Some or all of process 600 may be derived from... Figure 5 One or more components in process 600 may be executed as described herein. For example, some or all of process 600 may be executed by... Figure 1 , Figure 3 and / or Figure 5 One or more components in the process may perform the operation as described herein. For example, some or all of the process 600 may be performed by vehicle computing device 504 or computing device 534.

[0130] At operation 602, the process may include receiving first data indicating historical locations associated with multiple objects in the environment by a first machine learning model. In some examples, operation 602 may include a computing device implementing a prediction component 104 to receive log data associated with one or more autonomous vehicles. The log data may, for example, identify the previous locations of various objects of different object types within a previous time period. The prediction component 104 may also receive map data including features of the real-world environment and / or the simulated environment. In various examples, the log data and / or map data may be associated with previous navigation of the autonomous vehicle in the real-world environment and / or previous simulations in the simulated environment. In various instances, data input to the prediction component 104 (e.g., input data 308) may be received from storage devices or components of the vehicle's computing device.

[0131] In some examples, operation 602 may include receiving input data including a top-down representation 208 and / or feature vectors representing the environment and / or objects. For example, prediction component 104 may receive feature vector 226 from machine learning model 224 and / or feature vector 230 from machine learning model 228 as part of the input data. In some examples, the input data may represent historical data associated with one or more objects in the environment.

[0132] At operation 604, the process may include predicting a first set of candidate locations associated with the object by a first machine learning model and at least in part based on the first data. In some examples, operation 604 may include prediction component 104 implementing a machine learning model to determine a distribution of candidate object locations representing regions, points, or locations in the environment that the object may occupy in the future. In some examples, the first set of candidate locations may include at least a portion of the candidate object locations in the distribution. In some examples, prediction component 104 may select a subset of the candidate object locations from the distribution as the first set of candidate locations.

[0133] At operation 606, the process may include receiving second data indicating the location associated with an autonomous vehicle in the environment by a second machine learning model. In some examples, operation 606 may include a prediction component 104 implementing another machine learning model configured to identify a distribution of candidate vehicle locations representing areas, points, or locations in the environment that the vehicle may occupy in the future. For example, prediction component 104 may receive historical vehicle data, log data, etc., describing previous vehicle locations, and / or predicted vehicle locations describing future vehicle locations. In various examples, the second machine learning model may also, or alternatively, receive map data and / or planner data (e.g., trajectory data, location data, etc., from a self-planning component such as planning component 524). Although described as different machine learning models, in some examples, functionality associated with the first and second machine learning models may be combined into a single machine learning model.

[0134] At operation 608, the process may include predicting a second set of candidate locations associated with the autonomous vehicle by a second machine learning model and at least in part based on the second data. In some examples, operation 608 may include prediction component 104 determining a distribution or dataset of candidate vehicle locations representing areas, points, or locations in the environment that the autonomous vehicle (e.g., vehicle 102) may occupy in the future. In various examples, the second set of candidate locations associated with the autonomous vehicle may include locations associated with vehicle 102 or another vehicle (e.g., in a real-world or simulated environment).

[0135] The difference is compared with a difference threshold to determine whether the difference meets or exceeds the difference threshold.

[0136] Figure 6B This is the second part of a flowchart depicting an example process 600 for determining the location of an object using one or more example models.

[0137] At operation 610, the process may include clustering the first set of candidate locations associated with the object relative to a second set of candidate locations associated with the autonomous vehicle. In some examples, operation 610 may include clustering component 304 determining clusters (e.g., first cluster 410) of some candidate locations in the first set of candidate locations associated with the object based on candidate locations in the second set of candidate locations associated with the autonomous vehicle. Candidate locations in the first cluster, if occupied by an object, would cause the same vehicle behavior in the future. For example, the autonomous vehicle may brake (or take some other action) to avoid an object at a location in the cluster, regardless of the type of the object or the location from which the object originated before occupying the location. In various examples, additional clusters associated with different autonomous vehicle behaviors may be determined to capture a variety of potential vehicle behaviors for different candidate object locations.

[0138] At operation 612, the process may include third data, at least in part, based on clustering, to determine a subset representing a first set of candidate locations associated with an object. Such a subset may be based on, for example, ratings described in detail herein. In some examples, operation 612 may include prediction component 104 comparing locations within clusters to each other and / or to vehicle locations, and selecting candidate locations within the respective clusters as a subset of the first set of candidate locations.

[0139] At operation 614, the process may include determining whether the model is currently being trained or whether it has been previously trained. For example, operation 614 may include determining whether a machine learning model associated with prediction component 104 is being trained. If the model is not trained (e.g., "No" in operation 614), the process may proceed to operation 616 to transmit third data to a vehicle computing device associated with the autonomous vehicle, configured to determine actions for the autonomous vehicle to avoid objects. If the model is trained (e.g., "Yes" in operation 614), the process proceeds to operation 618 to update the model(s) parameters at least in part based on the model's output. Of course, in some examples, depending on the implementation, the operations may be performed in parallel.

[0140] At operation 616, the process may include sending third data to a vehicle computing device associated with the autonomous vehicle, the vehicle computing device being configured to determine the autonomous vehicle's navigating actions relative to an object. In various examples, the vehicle computing device is configured to determine the vehicle's trajectory based at least in part on the output. For example, output from model component 530 may be sent to perception component 522 or planning component 524, to name a few. In various examples, the vehicle computing device may control the vehicle's operations, such as planning component 524. The vehicle computing device may determine the vehicle trajectory based at least in part on a subset of the object's location (e.g., the third data), thereby improving vehicle safety by planning the likelihood that the object might behave unexpectedly at some point in time. Additional details regarding the use of one or more outputs from one or more models to control the vehicle are discussed throughout this disclosure.

[0141] In some examples, operation 616 may include planning component 524 using a subset of object positions output by prediction component 104 to determine candidate trajectories or other actions for the vehicle to avoid potential intersections between the object and the vehicle. In some examples, operation 616 may include vehicle computing device 504 controlling the vehicle in the environment in the future based at least on output data 310 from prediction component 104. In some examples, operation 616 may include controlling the vehicle in a real-world environment based at least in part on transmitted data.

[0142] At operation 618, the training component (e.g., training component 548) can update, modify, and / or augment one or more parameters of the machine learning model to train the model. In some instances, the output from model component 530 can be compared with training data (e.g., representing the true values ​​of labeled data) for training. Based at least in part on this comparison, training component 548 can identify parameters associated with model component 530 for updating. In various examples, the output can be used to train models such as convolutional neural networks and / or graph neural networks.

[0143] In various examples, process 600 may return to 602 after performing operations 616 and / or 618. In such examples, the vehicle may continuously monitor for potential collisions and update / modify decisions regarding whether to engage with the safety system (in at least some examples, this may include performing one or more maneuvering actions to mitigate or minimize a collision). In any of the examples described herein, the process may be repeated at a given frequency and generate one or more occupancy grids associated with one or more future times for the determinations described above.

[0144] Figure 6A and 6BExample processes according to this disclosure are illustrated. These processes are shown as logic flowcharts, where each operation represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents computer-executable instructions stored on one or more computer-readable storage media that perform the operation when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which operations are described is not intended to be construed as limiting, and any number of described operations can be omitted or combined in any order and / or in parallel to implement the process. In some embodiments, one or more operations of the method can be omitted entirely. By way of example and not limitation, operations 602, 604, and 610 can be performed without operations 606 and 608 (e.g., receiving a single vehicle position can be alternatively performed). Furthermore, the methods described herein can be combined with each other, in whole or in part, or with other methods.

[0145] The methods described herein represent sequences of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, a box represents computer-executable instructions stored on one or more computer-readable storage media that perform the operations when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as restrictive, and any number of the described operations can be omitted or combined in any order and / or in parallel to implement the process.

[0146] The various techniques described herein can be implemented in the context of computer-executable instructions or software, such as program modules, which are stored in a computer-readable storage device and executed by the processor of one or more computing devices, such as those shown in the accompanying drawings. Typically, program modules include routines, programs, objects, components, data structures, etc., and define operational logic for performing a specific task or implementing a specific abstract data type.

[0147] Other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific responsibility distributions have been defined above for discussion purposes, various functions and responsibilities may be distributed and divided in different ways depending on the circumstances.

[0148] Similarly, software can be stored and distributed in various ways and using different means, and the specific software storage and execution configuration described above can vary in many different ways. Therefore, software implementing the above techniques can be distributed across various types of computer-readable media, and is not limited to the form of the memory specifically described.

[0149] Exemplary terms Any exemplary terms in this section may be used in conjunction with any other exemplary terms and / or any other examples or embodiments described herein.

[0150] A: A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform actions including: receiving, by a first machine learning model: first data indicating historical locations associated with a plurality of objects in an environment; predicting, by the first machine learning model and at least in part based on the first data, a first set of candidate locations associated with the objects; receiving, by a second machine learning model: first data and second data indicating locations associated with an autonomous vehicle in the environment; predicting, by the second machine learning model and at least in part based on the first data and the second data: a second set of candidate locations associated with the autonomous vehicle; clustering, at least in part based on the second set of candidate locations associated with the autonomous vehicle, the first set of candidate locations associated with the objects; determining, at least in part based on the clustering, third data representing a subset of the first set of candidate locations associated with the objects; and transmitting the third data to a vehicle computing device associated with the autonomous vehicle, the vehicle computing device being configured to determine actions for navigating the autonomous vehicle relative to the objects.

[0151] B: According to the system described in paragraph A, the action further includes: inputting fourth data, including object state data or vehicle state data, into the first machine learning model, wherein the prediction of the first set of candidate locations associated with the object is also based at least in part on the fourth data.

[0152] C: According to the system described in paragraph A or B, the action further includes: determining a first cluster of at least some candidate positions in the first group of candidate positions associated with the object; and determining a second cluster of at least some candidate positions in the first group of candidate positions associated with the object, wherein determining the third data includes selecting a first candidate position from the first cluster and selecting a second candidate position from the second cluster.

[0153] D: According to the system described in paragraph C, the actions further include: generating a first reference action associated with a first candidate location in a first cluster and a second reference action associated with a second candidate location in a second cluster; and causing the vehicle computing device to determine the actions of the autonomous vehicle relative to the object navigation environment based at least in part on the first reference action or the second reference action.

[0154] E: The system according to any one of paragraphs A, D, and E, wherein: the first data represents the location information of the plurality of objects within a time period, and the second data includes at least one of the following: the previous location of the autonomous vehicle in a previous time period, or the predicted location associated with the predicted trajectory of the autonomous vehicle.

[0155] F: A method comprising: determining candidate object locations for an object based at least in part on historical data associated with a plurality of objects in an environment, the plurality of objects including the object; determining candidate vehicle locations for a vehicle to occupy in the future; clustering a first portion of the candidate object locations into a first cluster and a second portion of the candidate object locations into a second cluster based at least in part on the candidate vehicle locations of the vehicle; determining a subset of object locations from the candidate object locations based at least in part on the clusters; and transmitting the subset of object locations to a vehicle computing device to determine actions for the vehicle to navigate relative to the object.

[0156] G: The method described in paragraph F further includes: receiving map data including a top-down representation of the environment; wherein determining the locations of candidate objects and candidate vehicles is also based at least in part on the map data.

[0157] H: The method according to paragraph F or G further includes: receiving vehicle data indicating at least one of the following: a previous location of the vehicle in a previous time period or a predicted location associated with a predicted trajectory of the vehicle; and wherein determining candidate vehicle locations to be occupied by the vehicle in a future time period is based at least in part on the vehicle data.

[0158] I: The method according to any one of paragraphs FH, wherein: the first cluster is associated with a first vehicle behavior, and the second cluster is associated with a second vehicle behavior different from the first vehicle behavior.

[0159] J: The method according to any one of paragraphs FI further includes: generating a first reference action associated with a first candidate position in a first cluster and a second reference action associated with a second candidate position in a second cluster; and causing the vehicle computing device to determine the vehicle's actions at least in part based on the first reference action or the second reference action.

[0160] K: The method described in paragraph J further includes: associating a first reference action and a second reference action with nodes of a tree search; and causing the vehicle computing device to determine the vehicle's actions at least in part based on the tree search.

[0161] L: The method according to any one of paragraphs FK, wherein determining a subset of object locations from candidate object locations includes: determining a first probability that an object occupies a first candidate object location in a first cluster, and a second probability that an object occupies a second candidate object location in a first cluster; and determining a first candidate object location or a second candidate object location from the first cluster based at least in part on the first probability and the second probability, wherein the subset of object locations includes at least one of the following: a first candidate object location or a second candidate object location from the first cluster.

[0162] M: The method according to any one of paragraphs FL, wherein determining a subset of the object locations from the candidate object locations comprises: determining a cost associated with excluding the first cluster; and determining candidate object locations from the first cluster to include in the subset of the object locations based at least in part on the cost.

[0163] N: The method according to any one of paragraphs FM further includes: determining, at least in part, a threshold number of object locations to be included as a subset of object locations based on one or more criteria.

[0164] O: The method according to any one of paragraphs FN, wherein: a first score associated with a first cluster and a second score associated with a second cluster are determined; and a subset of object locations is determined from candidate object locations based at least in part on the first score and the second score.

[0165] P: The method according to any one of paragraphs FO, wherein the clustering is performed by a machine learning model trained to cluster the candidate object locations based at least in part on one of vehicle actions or vehicle positions.

[0166] Q: A non-transitory computer-readable medium storing one or more instructions that, when executed, cause one or more processors to perform actions, the actions comprising: determining candidate object locations for an object based at least in part on historical data associated with a plurality of objects in an environment, the plurality of objects including the object; determining candidate vehicle locations to be occupied by a vehicle in the future; clustering a first portion of the candidate object locations into a first cluster and a second portion of the candidate object locations into a second cluster based at least in part on the candidate vehicle locations of the vehicle; determining a subset of object locations from the candidate object locations based at least in part on the clusters; and transmitting the subset of object locations to a vehicle computing device to determine an action for navigating the vehicle relative to the object.

[0167] R: According to one or more non-transitory computer-readable media as described in paragraph Q, the action further includes: receiving map data including a top-down representation of the environment; and wherein determining the candidate object locations and the candidate vehicle locations is also based at least in part on the map data.

[0168] S: According to one or more non-transitory computer-readable media as described in paragraph Q or R, the action further includes: receiving vehicle data indicating at least one of the following: a previous location of the vehicle in a previous time period or a predicted location associated with a predicted trajectory of the vehicle, wherein the determination of the candidate vehicle location to be occupied by the vehicle in a future time period is based at least in part on the vehicle data.

[0169] T: One or more non-transitory computer-readable media for any item in paragraph QS, wherein: a first cluster is associated with a first vehicle behavior, and a second cluster is associated with a second vehicle behavior different from the first vehicle behavior.

[0170] While the exemplary clauses described below are for a particular implementation, it should be understood that, in the context of this document, the content of the exemplary clauses may also be implemented via methods, devices, systems, computer-readable media, and / or other implementations. Furthermore, any of instances A through T may be implemented individually or in combination with any other one or more of instances A through T.

[0171] in conclusion Although one or more examples of the techniques described herein have been described, various modifications, additions, substitutions, and equivalents thereof are included within the scope of the techniques described herein.

[0172] In the description of the examples, reference is made to the accompanying drawings, which form part of the description and illustrate specific examples of the claimed subject matter. It should be understood that other examples may be used, and changes or alterations, such as structural changes, may be made. Such examples, changes, or alterations do not necessarily deviate from the scope of the intended claimed subject matter. While the steps herein may be presented in a specific order, in some cases the order may be changed so that certain inputs are provided at different times or in a different order without altering the functionality of the described system and method. The disclosed processes may also be performed in different orders. Furthermore, the various calculations in this document do not need to be performed in the disclosed order, and other examples using alternative orders of calculations can be readily implemented. In addition to being reordered, calculations may also be decomposed into sub-computations with the same results.

Claims

1. A method comprising: Candidate object locations for an object are determined at least in part based on historical data associated with multiple objects in the environment, including the object itself. Determine the candidate vehicle positions that the vehicle will occupy in the future; At least in part based on the candidate vehicle locations of the vehicle, a first portion of the candidate object locations is clustered into a first cluster, and a second portion of the candidate object locations is clustered into a second cluster; A subset of object locations is determined from the candidate object locations, at least in part based on the clustering. as well as A subset of the object's location is transmitted to the vehicle's computing device to determine actions for the vehicle to navigate relative to the object.

2. The method according to claim 1, further comprising: Receive map data including a top-down representation of the environment. The determination of the candidate object location and the candidate vehicle location is also based, at least in part, on the map data.

3. The method according to any one of claims 1 or 2, further comprising: Receive vehicle data indicating at least one of the following: the vehicle's previous location within a previous time period, or the predicted location associated with the vehicle's predicted trajectory. The determination of the candidate vehicle location to be occupied by the vehicle in the future time is based at least in part on the vehicle data.

4. The method according to any one of claims 1-3, wherein: The first cluster is associated with a first vehicle behavior, and the second cluster is associated with a second vehicle behavior that is different from the first vehicle behavior.

5. The method according to any one of claims 1-4, further comprising: Generate a first reference action associated with a first candidate position in the first cluster, and a second reference action associated with a second candidate position in the second cluster; as well as This enables the vehicle computing device to determine the vehicle's actions at least in part based on the first reference action or the second reference action.

6. The method according to claim 5, further comprising: Associate the first reference action and the second reference action with the nodes of the tree search; as well as This enables the vehicle computing device to determine the vehicle's actions based at least in part on the tree search.

7. The method according to any one of claims 1-6, wherein, Determining a subset of the object locations from the candidate object locations includes: Determine a first probability that the object occupies a first candidate object position in the first cluster, and a second probability that the object occupies a second candidate object position in the first cluster; and The location of the first candidate object or the location of the second candidate object is determined from the first cluster, at least in part based on the first probability and the second probability. The subset of object locations includes at least one of the following: the first candidate object location or the second candidate object location from the first cluster.

8. The method according to any one of claims 1-7, wherein, Determining a subset of the object locations from the candidate object locations includes: Determine the costs associated with excluding the first cluster; and Candidate object locations are determined from the first cluster based at least in part on the cost, to be included in a subset of the object locations.

9. The method according to any one of claims 1-8, further comprising: The threshold number of object locations to be included as a subset of object locations is determined, based at least in part on one or more criteria.

10. A computer program product comprising encoded instructions, wherein when run on a computer, the encoded instructions implement the method according to any one of claims 1-9.

11. A system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein, when executed, the instructions cause the system to perform actions, including: Candidate object locations for an object are determined at least in part based on historical data associated with multiple objects in the environment, including the object itself. Determine the candidate vehicle positions that the vehicle will occupy in the future; At least in part based on the candidate vehicle locations of the vehicle, a first portion of the candidate object locations is clustered into a first cluster, and a second portion of the candidate object locations is clustered into a second cluster; At least in part based on the clustering, a subset of object locations is determined from the candidate object locations; and A subset of the object's location is transmitted to the vehicle's computing device to determine actions for the vehicle to navigate relative to the object.

12. The system according to claim 11, wherein the action further includes: Receive map data including a top-down representation of the environment. The determination of the candidate object location and the candidate vehicle location is also based, at least in part, on the map data.

13. The system according to any one of claims 11 or 12, wherein the action further comprises: Receive vehicle data indicating at least one of the following: the vehicle's previous location within a previous time period, or the predicted location associated with the vehicle's predicted trajectory. The determination of the candidate vehicle location to be occupied by the vehicle in the future is based at least in part on the vehicle data.

14. The system according to any one of claims 11-13, wherein: Determine a first score associated with the first cluster and a second score associated with the second cluster; and The subset of the object locations determined from the candidate object locations is based at least in part on the first score and the second score.

15. The system according to any one of claims 11-14, wherein, The clustering is performed by a machine learning model trained to cluster the candidate object locations based at least in part on one of the following: vehicle actions or vehicle locations.

Images