Encoding relative object information into node-edge features

Abstract

Techniques for determining a unified future of an object in an environment are discussed herein. The techniques may include determining a first feature associated with an object in the environment and a second feature associated with the environment, and updating a graph neural network (GNN) based on a position of the object in the environment to encode the first feature and the second feature into a graph node representing the object and to encode relative positions of additional objects in the environment into one or more edges connected to the node. The GNN may be decoded to determine a predicted position of the object at a future time. Additionally, a predicted trajectory of the object may be determined using the predicted positions of the object at various times.

Description

[Technical field]

[0001] The present disclosure relates to encoding relative object information into node-edge features. [Background technology]

[0002] This application claims priority to U.S. patent application Ser. No. 17 / 535357, filed November 24, 2021, and entitled "ENCODING RELATIVE OBJECT INFORMATION INTO NODE EDGE FEATURES," U.S. patent application Ser. No. 17 / 535382, filed November 24, 2021, and entitled "BOUNDARY AWARE TOP-DOWN PREDICTION," U.S. patent application Ser. No. 17 / 535396, filed November 24, 2021, and entitled "PREDICTION SAMPLING TECHNIQUES," and U.S. patent application Ser. No. 17 / 535418, filed November 24, 2021, and entitled "FOCUSING PREDICTION DISTRIBUTION OUTPUT FOR EFFICIENT SAMPLING," the entire contents of which are incorporated by reference herein for all purposes.

[0003] An autonomous vehicle may include various software-based systems, hardware-based systems, and / or controllers for guiding the vehicle through an environment. For example, the controller of an autonomous vehicle may use sensor systems, object perception and prediction systems, and route planning and optimization techniques to plan a path, determine a driving route, and guide the vehicle through an environment including static and dynamic objects. While traversing a congested area with other moving vehicles (autonomous or otherwise), moving people, stationary buildings, etc., an autonomous vehicle may receive and analyze data and make decisions to ensure the safety of passengers as well as surrounding people and objects. For example, while traversing an environment, an autonomous vehicle may use a combination of sensor data from various sensors about objects in the surrounding environment and map data representative of the surrounding environment to analyze the environment and determine how to control and navigate the vehicle in the environment. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features.

[0005] [Figure 1] FIG. 1 is a pictorial flow diagram of an exemplary process for ingesting sensor data, encoding features into nodes and edges of a graph neural network (GNN), and sampling predicted locations of objects from the distribution data output by the GNN. [Diagram 2] FIG. 1 is a pictorial flow diagram illustrating an example of a process for ingesting sensor data, encoding features into nodes and edges of a GNN, determining that a first predicted location of an object sampled from distribution data output by the GNN is outside a specified region, and determining a second predicted location of the object based on map data. [Figure 3A] 1 shows an example of an environment from a top-down view and a predicted trajectory of a vehicle within the environment determined using a first type of prediction. [Figure 3B] 1 shows an example of an environment from a top-down view and the predicted future of a vehicle within the environment determined using the second type of prediction. [Figure 4A] 1 shows an example of an environment from a top-down view and a predicted position determined using a first type of prediction. [Figure 4B] FIG. 2 illustrates an example environment from a top-down view and a comparison of predicted positions determined using a second type of prediction with predicted positions determined using a first type of prediction. [Figure 5A] FIG. 2 illustrates an example structure for determining a predicted position at a first time using a first type of prediction and a second type of prediction. [Figure 5B] FIG. 2 illustrates an example structure for determining a predicted position at a second time using a first type of prediction and a second type of prediction. [Figure 6] FIG. 1 is a block diagram illustrating an example architecture for encoding features into a GNN and sampling predicted locations of objects from the distribution data output by the GNN. [Figure 7] FIG. 1 is a block diagram illustrating an example of updating the nodes of a GNN and sampling the predicted positions of objects from the distribution data output by the GNN. [Figure 8A] An example process for ingesting sensor data, encoding features into nodes and edges of a GNN, and sampling. [Figure 8B] An example process for ingesting sensor data, encoding features into nodes and edges of a GNN, and sampling. [Figure 9] 1 is an example process for ingesting sensor data, encoding features into nodes and edges of a GNN, determining that a first predicted location of an object sampled from distribution data output by the GNN is outside a specified region, and determining a second predicted location of the object based on map data. [Figure 10]An exemplary process for ingesting sensor data, encoding features into nodes and edges of a GNN, determining a sampling technique for sampling distribution data output by the GNN, and controlling a vehicle based on predicted positions of objects sampled from the distribution data using the sampling technique. [Figure 11] An exemplary process for ingesting sensor data, encoding features into nodes and edges of a GNN, evaluating first distribution data output by the GNN to determine second distribution data including a subset of predicted locations included in the first distribution data that satisfy a criteria, and controlling a vehicle based on predicted locations of objects sampled from the second distribution data. [Figure 12] FIG. 1 is a block diagram of an example system for implementing the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Techniques for determining predicted future positions of objects in an environment are discussed herein. For example, the techniques described herein may include prediction and planning actions based on analysis of sensor data, map data, and / or objects determined in the environment. In some examples, the techniques may include determining and vectorizing elements of the environment from a feature map associated with the environment, as well as determining and vectorizing objects (also referred to herein as entities, agents, etc.) perceived in the environment, and representing the vectorized environmental elements and objects in a graph structure. In some examples, a computing device of a vehicle, such as an autonomous vehicle, and / or a computing device associated with the vehicle may generate and use a graph neural network (GNN) that includes a combination of vehicle nodes and / or object nodes. The GNN may also include an edge network that stores offset data (e.g., relative position, relative attitude, relative velocity, relative acceleration, relative size, etc.) between pairs of objects in the GNN. A GNN is a type of neural network that may operate on a graph structure. In various implementations, the GNN may be partially connected or fully connected with distinct edge features associated with distinct pairs of nodes in the GNN. Machine learning based inference operations, such as graph message passing, may be performed to update the state of the GNN, including updating node and / or edge features, based on internal inputs determined from the GNN itself and / or based on updated observations perceived by the autonomous vehicle in the environment. The updates to the GNN (also referred to as data output by the GNN) may represent a distribution of predicted future states of individual objects in the environment, and the autonomous vehicle may decode portions of the GNN and / or sample from the distribution output to determine predictions of positions, velocities, trajectories, and / or other updated predicted states of objects in the environment.

[0007] As discussed herein, the sensor data may be captured by sensors associated with the vehicle in the environment. In some examples, the environment may include one or more objects, such as, for example, a first object (e.g., an entity). The sensor data may be processed in various ways to determine features associated with the vehicle, the first object, and / or the environment. For example, first data representing a first state of the first object may be determined. In some examples, the first state may include, but is not limited to, an object pose, an object position, an object acceleration, an object velocity, an object size, an object type, a lighting condition of the object, and the like. In some examples, the first data representing the first state of the first object may be input to a first machine learning (ML) model and processed to determine output data, such as, for example, second data including a first feature associated with the first object. The first ML model may be configured as a recurrent neural network (RNN). In some examples, the second data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the object.

[0008] Additionally or alternatively, map data associated with the environment or a portion thereof that the vehicle is traversing may be determined and / or received. In some examples, the sensor data and / or map data may be processed to determine third data representing a field of view of the environment. In some examples, the third data may be represented in a plan view or a top-down view. In other examples, the third data may be represented in an elevation view or a side view. In some examples, the field of view of the environment may represent various elements in the vehicle's environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc. In some examples, the third data representing the field of view of the environment may be input to a second ML model and processed to determine output data, such as, for example, fourth data including second features associated with the environment. The second ML model may be configured as an RNN or a convolutional neural network (CNN). In some examples, the fourth data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the environment. Additionally or alternatively, the sensor data and / or map data may be processed to determine a feature map representing an area associated with the environment, and a second feature may be determined based on a current position of a first object in the environment and a corresponding position on the feature map, and the second feature may be sampled from the feature map at the current position of the object to provide a scene context feature relative to the position of the object.

[0009] Once a first feature associated with the object and a second feature associated with the environment have been determined, a GNN may be generated and / or updated. In some examples, the GNN may be generated by using the first feature and the second feature to determine a graph node representing the first object. Additionally or alternatively, the GNN may be updated by associating the first feature and the second feature with the graph node representing the first object. Additionally or alternatively, a second graph node of the GNN representing the vehicle may be determined and / or updated. Once two or more graph nodes of the GNN have been determined and / or updated, an edge connecting the second node to the second node may be determined. In some examples, the edge connecting the two graph nodes may be encoded with features associated with the objects represented by the nodes to each other. For example, a first edge connecting a first graph node associated with the first object and a second graph node associated with the vehicle may be determined. The first edge may be encoded with features associated with the first object (e.g., the first and second features) to the vehicle and / or features associated with the vehicle to the first object.

[0010] As previously described, the predicted positions associated with the graph nodes of the GNN and / or the distribution data associated with the graph nodes of the GNN may be determined based on updating the GNN and / or performing inference operations on the GNN. The distribution data may include a distribution of predicted positions of objects in the future (e.g., at variable time increments). In some examples, the GNN may output a single Gaussian distribution for each object in the environment. The GNN may be configured such that the predicted positions included in the distribution data of an object are determined based on features associated with the object (e.g., first and second features) and are relative to other objects in the environment, i.e., a first predicted position of a first object may not overlap or otherwise collide with a second predicted position of a second object in the same time step.

[0011] The predicted location of the object in the future (e.g., from one time step to the next time step) may be determined in various ways. In some examples, the predicted location of the object may be determined by sampling from the distribution data according to a sampling technique such as Gaussian sampling (e.g., following a sampling pattern based on a Gaussian distribution), maximum likelihood sampling (e.g., taking an average of the minimum, maximum, or mean value of the distribution), or overwrite sampling (e.g., selecting a particular location of the object that may or may not be included in the distribution). For example, a first predicted location of a first object may be determined by sampling from the first distribution data output by the GNN.

[0012] Once a predicted position of the object (e.g., a predicted position at time step 1) is determined, additional predicted positions in the future (e.g., a predicted position at time step 2) may be determined from the predicted positions and / or a trajectory of the object may be determined based on the predicted positions. For example, once a first predicted position is determined, a first graph node representing the first object may be updated to reflect a first feature associated with the object at the first predicted position and / or a second feature associated with the environment at the first predicted position. Thus, second distribution data including a second predicted position of the object (e.g., at time step 2) may be output by the GNN following a graph node update or inference operation. Because the first predicted position of the object is reflected by the updated graph node, the second distribution data may include a second predicted position from the first predicted position. The second distribution data may then be sampled to determine a second predicted position of the object. This process may be repeated any number of times to determine multiple predicted positions of the object at incremental time steps (e.g., 0.1 seconds, 0.5 seconds, 1 second, 2 seconds, etc.) up to any time in the future (e.g., 1 second, 2 seconds, 4 seconds, 8 seconds, etc.).

[0013] The predicted positions of the objects may be utilized to determine a predicted trajectory of the objects. For example, a predicted trajectory of a first object may be determined based on a starting position (also referred to as a current position) of the first object (at time step 0), a first predicted position of the first object (at time step 1), and a second predicted position of the first object (at time step 2). The predicted trajectory may be configured to track along the predicted positions of the objects as time steps increase.

[0014] Additionally, although only described with respect to a single object in an environment, the techniques described herein may be performed in parallel (e.g., on a parallel processing device such as a graphics processing unit (GPU)) such that distribution data is output for each object in the environment at a single time step and includes the predicted position of the object at the next time step relative to other predicted positions of other objects in the environment. This may allow a set of N futures to be determined for the environment over a period of time, where N may be any integer greater than 0, and each future may include a single possible trajectory for each object in the environment (e.g., a trajectory that does not overlap with other trajectories and / or that does not result in collisions between objects).

[0015] Additionally or alternatively, the distribution data may be evaluated to determine focused distribution data that includes a subset of predicted locations that meet a criterion, allowing for more efficient sampling of predicted locations (e.g., eliminating impossible real-world scenarios, influencing an object to behave in a particular manner, etc.). In some examples, the criterion may be based on a type of object, a characteristic of the object, the map data, the input data, etc. In some examples, the distribution data may include one or more predicted locations of an object that may not reflect the real-world behavior of the object. For example, one or more predicted locations of the distribution data may not conform to rules of the road (which may be reflected by the map data and / or the input data), such as, for example, stop signs, traffic lights, speed limits, construction zones, etc. Additionally or alternatively, it may be desirable to evaluate objects that behave in an adversarial manner (e.g., driving recklessly, turning out of an improper lane, making sudden stops, etc.). In some examples, the distribution data may include a limited number of predicted locations that reflect objects that behave in an adversarial manner, such that using traditional random sampling techniques would result in few selections of such predicted locations. By evaluating the distribution data with respect to criteria (e.g., enforcement of one or more rules of the road, behaving in a hostile manner, etc.), a subset of the distribution data may be determined that includes only predicted locations that meet the criteria.

[0016] Additionally or alternatively, the distribution data may be sampled using various sampling techniques. In some examples, a sampling technique for sampling a set of distributions may be determined based on a classification type of the graph node (e.g., whether the graph node is associated with a vehicle, an object, a particular type of object, etc.), a predicted location of the distribution data (e.g., whether the predicted location is within a predefined region), a distribution type of the distribution data, an input associated with the distribution data (e.g., a reference trajectory associated with the object), and the like. In some examples, different sampling techniques may be employed for the same object at different time steps. For example, a first predicted location of the object may be determined at a first time using a Gaussian sampling technique, and a second predicted location of the object may be determined at a second time using a sampling technique configured to maximize the progress of the object at the second time. Additionally or alternatively, different sampling techniques may be employed for different objects at the same time step. For example, a first predicted position for a first object may be determined at a first time using a first sampling technique configured to minimize progress of the first object at the first time, and a second predicted position for a second object may be determined at a first time using a second sampling technique configured to maximize progress of the second object at the first time. Additionally or alternatively, a different sampling technique may be employed to overwrite the distribution data output for a specified object of the plurality of objects. For example, the distribution output for the object may include a predicted position of the object, and an input may be received to configure a sampling technique to determine a predicted position of the object following the reference trajectory.

[0017] Additionally or alternatively, the distribution data may be further evaluated to determine whether the distribution data should be overwritten. Take for example a region of the environment including the vehicle and the first object (e.g., a fixed region surrounding the vehicle capturing sensor data). In some examples, the region is configured as a top-down scene associated with the vehicle and generated based on the sensor data. Additionally or alternatively, the region of the environment may be limited by a specified distance, a trajectory and / or speed associated with the vehicle, an area where the sensor can capture sensor data to determine environmental features, etc. In some examples, the distribution data of the first object may be evaluated to determine whether the predicted location of the first object, or a threshold number of predicted locations, is outside the region or inside the region.

[0018] When the first object leaves the region of the environment, the graph node representing the second feature associated with the environment may be updated with the null scene context data. In this way, the GNN outputs distribution data based on the features associated with the object and the null scene context features associated with the environment, resulting in a predicted position of the object that may not take into account the features of the environment. For example, the road along which the object travels may have turns or curves. In some instances, the distribution data of the object outside the region may include predicted positions that do not follow the curvature of the road, and repeated position predictions in such scenarios may result in an unrealistic predicted trajectory of the object off the road. Therefore, it may be beneficial to utilize map data related to the environment instead of features associated with the environment to determine the predicted positions of the object outside the region of the environment.

[0019] For example, it may be desirable to determine a predicted position of an object that fits into the environment outside a boundary region of the environment (e.g., when the scene context is null). In some examples, map data associated with the environment may be utilized to project a first predicted position sampled from the distribution data to a second predicted position, e.g., a center position of a lane in which the object is traveling. In some examples, a reference line may be determined based on the current position of the object and the map data. For example, a reference line from the current position of the object to a position in the lane may be determined based on a distance substantially similar to a distance from the current position to the predicted position sampled from the distribution data. Additionally or alternatively, the reference line may be modified based on a vehicle dynamics model. For example, the reference line may be shortened or lengthened based on a vehicle dynamics model configured to model real-world behavior of the object, e.g., decelerating around a curve, accelerating around a corner, etc.

[0020] The techniques discussed herein may improve the capabilities of computing devices, such as computing devices of autonomous vehicles, in many ways. For example, the various techniques described herein provide technical improvements in the environment modeling and prediction capabilities of autonomous vehicles, as well as technical advantages of reducing computing resources and improving the efficiency of prediction and decision-making components of autonomous vehicles by examining objects in an environment not in isolation (e.g., ignoring the interactions of other agents) but with the corresponding interactions of other objects in the environment. For example, determining a predicted position of an object using additional object features encoded for the object allows for determining N-sets of possible futures of all objects in the environment, requiring less central processing unit (CPU) and / or graphics processing unit (GPU) power than determining N-sets of predicted trajectories of each object that may be incompatible (e.g., trajectories that result in collisions). The techniques described herein may also allow for more efficient sampling of distribution data by focusing the distribution data to include only predicted positions that meet criteria. Additionally, the techniques may determine realistic trajectories of objects outside of a bounded region (e.g., following the curvature of a lane) without using scene context data. Furthermore, determining the future of objects in the environment may improve the functionality of subsequent subsystems (e.g., planning systems) that rely on such data for safe vehicle operation. Thus, the techniques described herein may not only improve the functionality of the computing device, but also improve safety outcomes.

[0021] The techniques described herein may be implemented in many ways. Exemplary implementations are provided below with reference to the following figures. Although described in the context of an autonomous vehicle, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In one example, similar techniques may be utilized in driver-controlled vehicles, where such systems may provide indications of whether it is safe to perform various maneuvers. In another example, the techniques may be utilized in manufacturing assembly line situations, aerial surveying situations, or marine situations. Additionally, the techniques described herein may be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0022] FIG. 1 is a pictorial flow diagram of an exemplary process 100 for ingesting sensor data, encoding features into nodes and edges of a graph neural network (GNN), and sampling predicted locations of objects from the distribution data output by the GNN.

[0023] At operation 102, process 100 may include capturing and / or receiving sensor data of a physical or simulated environment. Example 104 shows a vehicle 106 capturing sensor data 108 of the environment. In some examples, the sensor data 108 may include lidar data, radar data, sonar data, time-of-flight data, or other depth data. For example, operation 102 may include capturing image data and generating depth data based on the captured image data.

[0024] At operation 110, process 100 may include determining a first characteristic associated with an object in the environment. In some examples, operation 110 may include determining first data representing a state of the object in the environment. In some examples, the state may include, but is not limited to, an pose of the object, a position of the object, an acceleration of the object, a velocity of the object, a size of the object, a type of the object, a lighting condition of the object, etc.

[0025] In some examples, the object feature component associated with the vehicle 106 may determine a first feature associated with the object by processing first data representing a state of the object with a machine learning (ML) model. For example, the first data representing a state of the object may be input into the first ML model and processed to determine output data, such as, for example, second data including the first feature associated with the object. In some examples, the first ML model may be configured as a recurrent neural network (RNN). In some examples, the second data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the object. For example, the first feature may be configured as a feature vector including vectorized object elements encoded as a feature vector.

[0026] At operation 112, process 100 may include determining a second feature associated with the environment. In some examples, operation 112 may include determining third data representing a field of view of the environment. In some examples, the third data may be represented in a plan view or a top-down view. In other examples, the third data may be represented in an elevation view or a side view. The third data representing the field of view of the environment may be determined using the sensor data 108 and / or map data associated with the environment and received from a map data server. In some examples, the field of view of the environment may represent various elements in the vehicle's environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc.

[0027] In some examples, the environmental features component associated with the vehicle 106 may determine a second feature associated with the environment by processing third data representing a field of view of the environment with an ML model. For example, the third data representing a field of view of the environment may be input to the second ML model and processed to determine output data, such as, for example, fourth data including the second feature associated with the environment. The second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0028] The map data received in operation 112 may include data representing various map features (or map elements) in the autonomous vehicle's environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc. In some cases, the map data may depict these and other types of permanent or semi-permanent map elements (e.g., road closures, road damage, construction sites, accidents, etc.), but may not include temporary objects such as other vehicles, bicycles, pedestrians in the environment, or temporary road features such as disabled vehicles, road hazards, short-term construction projects, etc. In some examples, the fourth data is configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the environment. For example, the second feature may be configured as a feature vector that includes one or more vectorized environment and / or map elements encoded as a feature vector.

[0029] Example 114 illustrates sensor data 108 captured by vehicle 106 and used to determine object features 116, and map data 118 associated with the environment used to determine environmental features 120. In some examples, a first feature may correspond to object feature 116 and a second feature may correspond to environmental feature 120. In some examples, a portion of sensor data 108 corresponding to an object may be processed to determine object feature 116. In some examples, environmental features 120 may be determined using sensor data 108 and map data 118. For example, sensor data 108 and / or map data 118 may be processed to determine a feature map representing the environment, a second feature may be determined based on a current location of the object in the environment and a representing location on the feature map, and the second feature may be sampled from the feature map at the current location of the object to provide a scene context feature for the location of the object.

[0030] At operation 122, the process 100 may include determining a first node of a graph neural network (GNN) representing the first feature (e.g., the object feature 116) and the second feature (e.g., the environment feature 120). At operation 122, the vehicle 106 may generate and / or update the GNN (or other graph structure) to include the object feature 116 and / or the environment feature 120 associated with the first node representing the object.

[0031] In some cases, the modeling component associated with the vehicle 106 may receive vectorized representations of objects (e.g., map elements and / or objects) from the object feature component and / or the environmental feature component and may create new nodes in the GNN, remove nodes from the GNN, and / or modify existing nodes of the GNN based on the received map data and / or entity data. Additionally, the modeling component may create and maintain edge features associated with node pairs in the GNN. As described above, nodes of a GNN may store a set of attributes that represent objects, and edge features may include data that indicates relative information (e.g., position, pose, etc.) of pairs of nodes.

[0032] At operation 124, the process 100 may include determining an edge connecting the first node and a second node of the GNN representing the vehicle 106. In some examples, the edge may represent the position of an object or vehicle as a relative position. In some examples, the edge may be configured as a bidirectional edge of the GNN. That is, with respect to the first node, the edge may represent the relative position of a target (e.g., the vehicle 106) with respect to a source (e.g., the object). Additionally or alternatively, with respect to the second node, the edge may represent the relative position of a target (e.g., the object) with respect to a source (e.g., the vehicle 106).

[0033] As mentioned above, edges in a GNN may be associated with pairs of nodes, and edge data (or edge features) associated with the edges may include relative data between the source and target nodes of the node pair. As an example, an edge connecting a first node representing an object and a second node representing a vehicle may store or have associated edge data including the relative distance, relative yaw, relative velocity, relative attitude, relative size, relative acceleration, relative tolerance, etc. between the first node and the second node. In some examples, the GNN is fully connected, with a different edge associated with each pair of nodes. In other examples, the GNN is partially connected, with not every different pair of nodes having an associated edge.

[0034] The example 126 shows an environment 128 including a vehicle 106, an object 130, a first node 132 associated with the object 130, a second node 134 associated with the vehicle 106, and an edge 136 connecting the first node 132 and the second node 134. Although not depicted in FIG. 1 for clarity of illustration, in some examples, the GNN may be a fully connected structure in which each unique pair of nodes is associated with a unique edge feature and / or edge data. Additionally or alternatively, the GNN may include additional nodes for each object in the environment and / or additional edges connecting the additional nodes. The process of updating the GNN nodes and / or edges between time steps is described in more detail with respect to FIG. 7.

[0035] At operation 138, the process 100 may include determining an output representing distribution data including a first predicted position of the object 130 in the future. In some examples, an inference operation may be performed to update node states and / or edge features of the GNN. Additionally or alternatively, a distribution component associated with the vehicle 106 may determine distribution data representing the predicted position of the object 130 based on the updated GNN. Additionally or alternatively, the output may represent a predicted position of the object 130 in the future.

[0036] As described above, the graph structure of the GNN includes nodes that represent features associated with the state of the object 130 and / or features related to the map elements associated with the object 130, which in combination represent the current state of the environment 128. In some examples, the GNN may be a completely stateless structure that may be generated anew each time the GNN generation process is executed based on data perceived by the autonomous vehicle 106 and / or map data received by the autonomous vehicle 106 from a map server.

[0037] Additionally or alternatively, the GNN may fully or partially retain its state during subsequent GNN generation processes, where certain object node, attribute, and / or edge data may be retained from a previous version of the GNN, while other data may be updated. For example, objects in the environment that were previously perceived by the autonomous vehicle 106, but may have moved to a predicted location at a future time, may be retained in the GNN and / or updated based on predictive data determined from the previous GNN.

[0038] The modeling component may support inference operations and functions based on machine learning techniques to update node and / or edge features to represent a predicted future state of the environment 128. For example, the inference operations performed by the modeling component to update the state of the GNN may be based entirely on the current state of the GNN. The modeling component may infer that an object 130 in the environment is likely to proceed to a predicted position based on its current speed and trajectory, within the constraints of the map elements, and based on the influence of other objects in the vicinity of the object 130.

[0039] As described herein, the inference operation may use machine learning techniques (e.g., trained based on driving logs and / or other training data) to determine a predicted future state of the GNN based on the current state of the GNN. The predicted future state of the GNN may correspond to updated object positions, velocities, trajectories, intent, and / or interactions that may occur between objects in the environment. Furthermore, within the environment represented by the GNN, the predicted future position and / or state of one object is often related to the predicted future position and / or state of other objects, and thus, a single execution of an inference process on the GNN may provide updated node states (representing inferred future positions, velocities, trajectories, etc.) for multiple (e.g., all) object nodes in the GNN. For such inference processes and other updates to the GNN that change the states of the nodes, the modeling component may also perform corresponding updates to edge features connected to those nodes, such that the updated edge features store accurate relative information based on the nodes associated with those edge features.

[0040] As described below, updates to the nodes and edges of the GNN may be based on inference operations performed by the GNN and / or based on new observations perceived by the autonomous vehicle. For example, after the autonomous vehicle initially perceives an entity in the environment at a first position, first velocity, first trajectory, etc., the autonomous vehicle may perceive the entity at a different time, at a different position, velocity, trajectory, etc. In some cases, the autonomous vehicle may perceive attributes (e.g., location, size, etc.) of a map element that differ from the map data received from the map server. Updated observations perceived by the autonomous vehicle are provided to the modeling component and used to update corresponding nodes of the GNN and, if necessary, to propagate the updates through edge features and / or other nodes of the GNN.

[0041] To determine a predicted state of the object at a future time, a distribution component associated with the autonomous vehicle may obtain node data and / or edge feature data from the updated GNN and decode the data to determine distribution data representing a predicted location of the object at a future time. The predicted location and / or state of the object may include, but is not limited to, a predicted position, velocity, acceleration, trajectory, yaw, attitude, or other attributes of the object in the environment. In some implementations, the component need not decode the entire GNN, but may select specific entities and sample / decode the corresponding node data and edge features to perform state predictions for the selected objects.

[0042] Example 140 illustrates a neural network 142 and a neural network output 144 that includes a predicted location distribution 146 for object 130. In some examples, neural network 142 may be configured as a GNN, and neural network output 144 may be configured as an output from the GNN. The predicted location distribution 146 included in neural network output 144 is described in more detail with respect to FIG.

[0043] In some examples, the distribution component may evaluate the distribution data 144 to determine focused distribution data that includes a subset of predicted locations 146 that meet criteria, allowing for more efficient sampling of predicted locations 146 (e.g., eliminating impossible real-world scenarios, influencing the object 130 to behave in a particular manner, etc.). In some examples, the criteria may be based on the type of object 130, characteristics of the object 130, map data, input data, etc. In some examples, the distribution data 144 may include one or more predicted locations 146 of the object 130 that may not reflect the real-world behavior of the object 130. For example, one or more predicted locations 146 of the distribution data 144 may not conform to rules of the road (which may be reflected by the map data and / or input data), such as, for example, stop signs, traffic lights, speed limits, construction zones, etc. Additionally or alternatively, it may be desirable to evaluate objects 130 that behave in an adverse manner (e.g., driving recklessly, swerving out of an improper lane, stopping suddenly, etc.). In some examples, the distribution data 144 may include a limited number of predicted locations 146 reflecting objects behaving in a hostile manner such that traditional random sampling techniques would be unlikely to select such predicted locations 146. By evaluating the distribution data 144 with respect to criteria (e.g., enforcing one or more rules of the road, engaging in hostile manners, etc.), a subset of the distribution data 144 may be determined that includes only those predicted locations 146 that satisfy the criteria.

[0044] At operation 148, the process 100 may include determining a predicted position of the object 130 in the future. In some examples, the predicted position of the object may be determined based on sampling from the distribution data 146.

[0045] The predicted position of the object 130 in the future (e.g., from one time step to the next) may be determined in various ways. In some examples, the predicted position of the object 130 may be output by the GNN following an inference operation and / or an update to the GNN. In some examples, the predicted position of the object 130 may be determined by sampling from the predicted position distribution 146 included in the neural network output 144 according to a sampling technique such as, for example, random sampling, Gaussian sampling (e.g., following a sampling pattern based on a Gaussian distribution), maximum likelihood sampling (e.g., taking an average of the minimum, maximum, or mean value of the distribution), or overwrite sampling (e.g., selecting a particular position of the object 130 that may or may not be included in the distribution). For example, a first predicted position of the object 130 may be determined by sampling from the predicted position distribution 146 output by the GNN.

[0046] Once a predicted position of the object 130 (e.g., a predicted position at time step 1) is determined, additional predicted positions in the future (e.g., a predicted position at time step 2) may be determined from the predicted positions. For example, once a first predicted position is determined, the first graph node 132 representing the object 130 may be updated to reflect a first feature associated with the object 130 at the first predicted position and / or a second feature associated with the environment 128 at the first predicted position. Thus, a second predicted position distribution 146 including a second predicted position of the object 130 (e.g., at time step 2) may be output by the GNN following an update and / or inference operation of the first graph node 132. Because the first predicted position of the object 130 is reflected by the updated graph node, the second predicted position distribution 146 may include a second predicted position from the first predicted position. The second predicted position distribution 146 may then be sampled to determine a second predicted position of the object 130. This process may be repeated any number of times to determine a number of predicted positions of object 130 at incremental time steps.

[0047] Additionally or alternatively, the predicted position distribution 146 may be sampled using various sampling techniques. In some examples, the sampling technique for sampling the predicted position distribution 146 may be determined based on a classification type of the graph node (e.g., whether the graph node is associated with a vehicle 106, an object 130, a particular type of object, etc.), a predicted position of the predicted position distribution 146 (e.g., whether the predicted position is within a predefined region), a distribution type of the predicted position distribution 146, an input associated with the predicted position distribution 146 (e.g., a reference trajectory associated with the object), etc.

[0048] In some examples, different sampling techniques may be employed for the same object 130 at different time steps. For example, a first predicted position of the object 130 may be determined at a first time using a Gaussian sampling technique, and a second predicted position of the object 130 may be determined at a second time using a sampling technique configured to maximize the progress of the object 130 at the second time.

[0049] Additionally or alternatively, different sampling techniques may be employed for different objects at the same time step. For example, a first predicted position for a first object (e.g., object 130) may be determined at a first time using a first sampling technique configured to minimize the progress of the first object at the first time, and a second predicted position for a second object (e.g., vehicle 106 or an additional object in environment 128) may be determined at the first time using a second sampling technique configured to maximize the progress of the second object at the first time.

[0050] Additionally or alternatively, a different sampling technique may be employed to overwrite the predicted position distribution 146 output by the GNN for a specified object of the plurality of objects. For example, the predicted position distribution 146 for the object 130 may include a predicted position of the object 130, and the sampling technique may receive an input including a reference trajectory to determine a predicted position of the object 130 following the reference trajectory. Additionally or alternatively, an additional predicted position distribution for the vehicle 106 may include a predicted position for the vehicle 106, and the input including the reference trajectory may be received from a planning component of the vehicle 106 to configure the sampling technique to determine a predicted position of the vehicle 106 following the reference trajectory.

[0051] At operation 150, process 100 may include determining a predicted trajectory of object 130. In some examples, the predicted trajectory of object 130 may be based on a predicted position of object 130. Additionally or alternatively, at operation 150, process 100 may include controlling vehicle 106 based on the predicted trajectory of object 130. In some examples, operation 150 may include generating a path, trajectory, and / or control signals for one or more systems of vehicle 106 to navigate vehicle 106 within the environment according to the predicted trajectory of object 130 and / or additional predicted trajectories associated with additional objects within the environment.

[0052] Example 152 shows an environment 128 including a vehicle 106, an object 130, and a predicted trajectory 156 of the object 130 determined based on one or more predicted positions of the object 130 sampled from one or more predicted position distributions 146 using one or more sampling techniques 154 described herein.

[0053] FIG. 2 is a pictorial flow diagram of an example process 200 for taking sensor data, encoding features into nodes and edges of a GNN, determining that a first predicted location of an object sampled from distribution data output by the GNN is outside a specified region, and determining a second predicted location of the object based on map data.

[0054] At operation 202, process 200 may include capturing and / or receiving sensor data of an environment. Example 204 shows a vehicle 206 capturing sensor data 208 of an environment. In some examples, the sensor data 208 may include lidar data, radar data, sonar data, time-of-flight data, or other depth data. For example, operation 202 may include capturing image data and generating depth data based on the captured image data.

[0055] At operation 210, process 200 may include determining a first characteristic associated with an object in the environment. In some examples, operation 210 may include determining first data representing a state of the object in the environment. In some examples, the state may include, but is not limited to, an pose of the object, a position of the object, an acceleration of the object, a velocity of the object, a size of the object, a type of the object, a lighting condition of the object, etc.

[0056] In some examples, in operation 210, an object feature component associated with the vehicle 206 may determine a first feature associated with the object by processing first data representing a state of the object with a machine learning (ML) model. For example, the first data representing a state of the object may be input into the first ML model and processed to determine output data, such as, for example, second data including the first feature associated with the object. In some examples, the first ML model may be configured as a recurrent neural network (RNN). In some examples, the second data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the object. For example, the first feature may be configured as a feature vector including vectorized object elements encoded as a feature vector.

[0057] At operation 212, process 200 may include determining a second feature associated with the environment. In some examples, operation 212 may include determining third data representing a field of view of the environment. In some examples, the third data may be represented in a plan view or a top-down view. In other examples, the third data may be represented in an elevation view or a side view. The third data representing the field of view of the environment may be determined using the sensor data 208 and / or map data associated with the environment and received from a map data server. In some examples, the field of view of the environment may represent various elements in the vehicle's environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc.

[0058] In some examples, the environmental features component associated with the vehicle 206 may determine a second feature associated with the environment by processing third data representing a field of view of the environment with an ML model. For example, the third data representing a field of view of the environment may be input to the second ML model and processed to determine output data, such as, for example, fourth data including the second feature associated with the environment. The second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0059] The map data received in operation 212 may include data representing various map features (or map elements) in the autonomous vehicle's environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc. In some cases, the map data may depict these and other types of permanent or semi-permanent map elements (e.g., road closures, road damage, construction sites, accidents, etc.), but may not include temporary objects such as other vehicles, bicycles, pedestrians in the environment, or temporary road features such as disabled vehicles, road hazards, short-term construction projects, etc. In some examples, the fourth data is configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the environment. For example, the second feature may be configured as a feature vector that includes one or more vectorized environment and / or map elements encoded as a feature vector.

[0060] Example 214 illustrates sensor data 208 captured by vehicle 206 and used to determine object features 216, and a feature map 218 associated with an area of ​​the environment that is used to determine environmental features 220. In some examples, a first feature corresponds to object feature 216 and a second feature corresponds to environmental feature 220. In some examples, a portion of sensor data 208 corresponding to an object may be processed to determine object feature 216. In some examples, environmental features 220 may be determined using sensor data 208 and map data. For example, sensor data 208 and / or map data may be processed to determine feature map 218 representing an area in the environment, a second feature may be determined based on a current location of an object in the environment and a corresponding location on feature map 218, and the second feature may be sampled from feature map 218 at the current location of the object to provide a scene context feature for the location of the object.

[0061] At operation 222, the process 200 may include associating the first feature (e.g., the object feature 216) and the second feature (e.g., the environment feature 220) with a node of a graph structure. In some examples, the graph structure may be configured as a graph structure, a GNN, and / or a neural network 142 as described with respect to FIG. 1. At operation 222, the vehicle 206 may generate and / or update the graph structure to include the object feature 216 and / or the environment feature 220 associated with the first node representing the object.

[0062] In some cases, the modeling component associated with vehicle 206 may receive vectorized representations of objects (e.g., map elements and / or objects) from the object feature component and / or the environmental feature component, and may create new nodes in the graph structure, remove nodes from the graph structure, and / or modify existing nodes in the graph structure based on the received map and / or object data. Additionally, the modeling component may create and maintain edge features associated with node pairs in the graph structure. As described above, nodes of the graph structure may store a set of attributes that represent objects, and edge features may include data indicating relative information (e.g., position, pose, etc.) of pairs of nodes.

[0063] Example 224 illustrates an example graph structure 226 of a neural network, such as a GNN. The graph structure 226 may include one or more nodes, including a node 228 associated with an object, and one or more edges connecting the nodes. Although not depicted in FIG. 2 for clarity of illustration, in some examples the graph structure may be a fully connected structure in which each unique pair of nodes is associated with a unique edge feature and / or edge data. Additionally or alternatively, the graph structure may include additional nodes for each object in the environment and / or additional edges connecting the additional nodes. The process of updating the graph nodes and / or edges between time steps is described in more detail with respect to FIG. 7.

[0064] At operation 230, the process 200 may include determining an output representing distribution data including a predicted location of the object. In some examples, the output representing the distribution data may be based at least in part on the graph structure. For example, an inference operation may be performed to update node states and / or edge features of the graph structure. Additionally or alternatively, a distribution component associated with the vehicle 206 may determine distribution data representing a predicted location of the object based on the updated GNN. Additionally or alternatively, the output may represent a predicted location of the object in the future.

[0065] At operation 232, the process 200 may include evaluating the distribution data to determine that the first predicted location of the object is outside the region of the environment. Additionally or alternatively, the distribution data of the object may be evaluated to determine whether the predicted location of the object, or a threshold number of predicted locations, is outside the region or inside the region. In some examples, the region of the environment may correspond to the region of the environment represented by the feature map 218. That is, nodes associated with objects in the environment region and / or edges connecting such nodes may be updated based on the environment features 220 sampled from locations on the feature map 216 corresponding to the location of the object in the environment. In some examples, the region may be configured as a top-down scene associated with the vehicle 206 and generated based on the sensor data 208. Additionally or alternatively, the region of the environment may be limited by a specified distance, a trajectory and / or speed associated with the vehicle 206, an area in which sensors can capture sensor data and determine environmental features, etc.

[0066] Example 234 illustrates a graph structure output 236 including an environment 238, a portion of the region of environment 240, an object 242, and distribution data 244 (shown overlaid on the environment for clarity) indicating a predicted position of the object 242, including a first predicted position 246 of the object 242. If the predicted position of the object 242 is outside the region of the environment 240, a graph node 228 associated with the object 242 may be updated based on the position of the object 242 relative to the feature map 218. As previously mentioned, the feature map 218 may be associated with a region of the environment 240 and may have null scene context data for regions outside the environment. In this manner, the graph node 228 may be updated to replace the environment features 220 with the null scene context data. At a subsequent time step, the graph structure 226 outputs distribution data based on features associated with the object 242 and null scene context features associated with the environment 238, resulting in a predicted position of the object that may not take into account the features of the environment 238.

[0067] At operation 246, the process 200 may include using the map data to determine a second predicted location of the object 242. In some examples, the second predicted location of the object 242 may be based in part on the first predicted location 246 sampled from the predicted location distribution 244 output by the graph structure 226.

[0068] Example 248 shows that the road in environment 238 along which object 242 travels has turns or curves. In some examples, subsequent distribution data for object 242 outside the region may include predicted positions (e.g., first predicted position 246) that do not follow the curvature of the road, and iteratively predicting positions in such a scenario may result in an unrealistic predicted trajectory of object 242 off the road. Thus, using map data to determine the predicted position of object 242 outside the region of environment 240 instead of environmental features 220 associated with environment 238 may be more accurate.

[0069] For example, using map data outside the bounded region of the environment 240 (e.g., when the scene context is null) to determine the predicted position and / or trajectory of the object 242 may result in a more accurate prediction. In some examples, the map data may be utilized to project the first predicted position 246 sampled from the distribution data 244 to a second predicted position 250, e.g., a center position of the lane in which the object 242 is traveling. In some examples, a baseline may be determined based on the current position of the object 242 and the map data. For example, the baseline from the current position of the object 242 to a position in the lane (e.g., the second predicted position 250) may be determined based on a distance substantially similar to the distance from the current position of the object 242 to the first predicted position 246 sampled from the distribution data 244. Additionally or alternatively, the baseline may be modified based on a vehicle dynamics model, such as, for example, the baseline distance from the current position to the predicted position. For example, the baseline distance may be shortened or lengthened based on a vehicle dynamics model configured to model real-world behavior of object 242, such as decelerating around curves, accelerating out of turns, etc.

[0070] In some examples, operation 246 may include determining a predicted trajectory of object 242. In some examples, the predicted trajectory of object 242 may be based on second predicted position 250 of object 242. Additionally or alternatively, in operation 246, process 200 may include controlling vehicle 206 based on second predicted position 250 and / or the predicted trajectory of object 242. In some examples, operation 246 may include generating a path, trajectory, and / or control signals for one or more systems of vehicle 206 to navigate vehicle 206 within environment 238 according to the predicted trajectory of object 242 and / or additional predicted trajectories associated with additional objects within environment 238.

[0071] 3A illustrates an example 300 that includes an environment 302(1) from a top-down view and predicted trajectories of vehicles (e.g., vehicle A 304, vehicle B 306, and / or vehicle C 308) within the environment 302(1) determined using a first type of prediction. The predicted trajectories for each of vehicles 304, 306, and 308 may be identified using key 312.

[0072] As shown in example 300, each vehicle 304, 306, and 308 (also referred to herein as objects, entities, agents, etc.) may have one or more predicted trajectories that are determined independently of one another. In some examples, the predicted trajectories in example 300 may be determined using only features associated with the nodes representing the particular vehicles 304, 306, and 308, and without using information encoded in edge features between the nodes that represent relative information of additional vehicles in environment 302(1). As can be seen in FIG. 3A, many of the trajectories of vehicles 304, 306, and 308 overlap, potentially resulting in collisions and / or unrealistic futures for environment 302.

[0073] 3B illustrates an example 310 including environments 302(2)-(N) from a top-down view and predicted futures (e.g., predicted futures 1-N, where N is any integer equal to or greater than 1) for vehicles (e.g., vehicle A 304, vehicle B 306, and / or vehicle C 308) within environments 302(2)-(N) determined using a second type of prediction. The predicted trajectories for each of vehicles 304, 306, and 308 in their respective futures may be identified using key 312.

[0074] As shown in example 310, each predicted future may include a single predicted trajectory for each vehicle 304, 306, 308 in the environment 302(2)-(N), determined relative to one another. In some examples, these predicted futures may be determined using features associated with nodes representing particular vehicles 304, 306, and 308, and information encoded in edge features between the nodes that represent relative information of additional vehicles in the environment 302(2)-(N). As can be seen in FIG. 3B, each future includes possible trajectories (e.g., collision-free trajectories) for each of the vehicles 304, 306, 308.

[0075] For example, consider environment 302(3) including predicted future 2 in example 310. In predicted future 2, vehicle A 304 has a predicted trajectory of making a left turn across an oncoming lane, determined using predicted positions of vehicle A 304 at various incremental time steps (e.g., 8 predicted positions determined over 8 seconds, one per second), such as predicted positions output by a GNN corresponding to the GNN described with respect to FIG. 1. During the update process of the graph nodes associated with vehicles 304, 306, 308, the predicted position distributions output for vehicle B 306 and / or vehicle C 308 may be based on the predicted positions of vehicle A 304. For example, the edges between vehicle B 306 and vehicle A 304 and between vehicle C 308 and vehicle A 304 may include information representative of the state of vehicle A 304, such as the state of a turn signal of vehicle A 304 indicating that vehicle A 304 is about to turn, and the acceleration of vehicle A 304 indicating that vehicle A 304 is about to begin a turn in the next time step. In this manner, the distributions output for vehicle B 306 and / or vehicle C 308 may include a predicted position taking into account the turn of vehicle A 304, to avoid unrealistic predicted positions and / or collisions of vehicle B 306 and / or vehicle C 308.

[0076] 3B illustrates four predicted futures in environment 302(2)-(N), any number of futures (e.g., tens, hundreds, thousands, etc.) may be determined. Additionally or alternatively, processing may be performed in parallel, such as on one or more graphics processing units (GPUs), to determine the futures substantially in parallel.

[0077] FIG. 4A illustrates an example environment 400 from a top-down view, including a road 402 (and a center position of the road 404) on which a vehicle 406 is traveling. The environment 400 may also include a boundary region 408 (e.g., a region corresponding to an autonomous vehicle as described with respect to FIG. 2) and a predicted position 410 determined using a first type of prediction. In some examples, the first predicted position 410 may correspond to a first predicted position 246 sampled from the distribution data 244 as described with respect to FIG. 2. Additionally or alternatively, the first type of prediction may correspond to determining a first predicted position 246 sampled from the distribution data 244 output by a graph structure as described with respect to FIG. 2. In some examples, the first predicted position 410 may be determined as a result of a predicted position of the vehicle 406 exiting the region 408, such that a node of the graph structure associated with the vehicle 406 may be updated with null scene context data.

[0078] 4B illustrates an example environment 420 from a top-down view, including a road 402 (and a center location of the road 404) on which a vehicle 406 is traveling. The environment 400 may also include a boundary region 408 (e.g., a region corresponding to an autonomous vehicle as described with respect to FIG. 2).

[0079] 2, if the predicted position of the vehicle 406 is outside the region 408, the graph node associated with the vehicle 406 may be updated based on the position of the vehicle 406 relative to the feature map associated with the region 408. That is, the feature map may be associated with the boundary region 408 and have null scene context data for the region of the environment 420 outside the boundary region 408. In this manner, the graph node may be updated and the scene context data from the previous time step may be replaced with the null scene context data. In subsequent time steps, the graph structure outputs distribution data based on the features associated with the vehicle 406 and the null scene context features related to the environment 420, resulting in a predicted position of the vehicle 406 that may not take into account features of the environment 420, such as the roads 402.

[0080] 4B, a first predicted position 410 from time step 0 (e.g., the current position of the vehicle 406) to time step 1 (e.g., the first predicted position 410) may be determined by sampling from the distribution output from the graph structure. As can be seen, the first predicted position 410 may position the vehicle 406 off and across the road 402 in the environment 420, which may result in an unrealistic trajectory of the vehicle 406.

[0081] When the first predicted location 410 is determined to be outside the region 408, the first predicted location 422 may be determined. In some examples, the first predicted location 422 may be determined using map data. For example, by using map data, the first predicted location 410 may be projected toward a center location of the road 404 as the first predicted location 422. In some examples, a reference line may be determined based on the current location of the vehicle 406 and the map data. For example, the reference line from the current location of the vehicle 406 to the center location of the road 404 may be determined based on a distance substantially similar to the distance from the current location of the vehicle 406 to the first predicted location 410 sampled from the distribution data.

[0082] Continuing this process, the graph structure may be updated with the first predicted position 422 and a second predicted position 424 may be sampled from the distribution of predicted positions of the vehicle 406 output by the graph structure. Again utilizing map data, the second predicted position 424 may be projected to a center position of the road 404 as the second predicted position 426 using a reference line. Additionally or alternatively, the reference line may be modified based on a vehicle dynamics model. For example, the reference line may be shortened or lengthened based on a vehicle dynamics model configured to model real-world behavior of the vehicle 406, e.g., decelerating around curves, accelerating out of turns, etc. For example, the reference line used to determine the second predicted position 426 may be shortened, such that the second predicted position 426 is closer to the first predicted position 422 than the second predicted position 424. Additionally or alternatively, the reference line used to determine the third predicted position 430 may be extended based on a third predicted position 428 sampled from the distribution of predicted positions output by the graph structure, such that the third predicted position 430 is a greater distance from the second predicted position 426 than the third predicted position 428 because the vehicle dynamics model indicates that the vehicle 406 is likely to accelerate on this portion of the curve due to the curvature of the road.

[0083] 5A shows an example structural diagram for determining a predicted position at a first time using a first type prediction and a second type prediction. A selected pose (e.g., an object for which the prediction is being performed) and information on which the selected pose depends (e.g., information and / or object used to determine the prediction) may be determined using key 502. In some examples, the first type prediction may correspond to the first type prediction as described with respect to FIG. 3A and / or the second type prediction may correspond to the second type prediction as described with respect to FIG. 3B.

[0084] As illustrated, determining a predicted position of entity 1 in time horizon 1 using the first type of prediction is determined independently from information associated with the vehicle and / or entity 2 in any given time horizon 1. That is, using the first type of prediction, a predicted position of entity 1 in time horizon 1 may be based solely on information associated with entity 1 in time horizon 1.

[0085] Alternatively, determining the predicted position of entity 1 in time horizon 1 using the second type of prediction is determined relative to information of all entities from all previous time horizons. That is, using the second type of prediction, the predicted position of entity 1 in time horizon 1 may be based on information associated with the vehicle, entity 1, and entity 2 in time horizon 0. In this manner, the predicted position for entity 1 may be based on a previous position of entity 1 in time horizon 0 and the respective positions of the vehicle and entity 2 in time horizon 0, and one or more predicted futures may be determined for the environment including the vehicle, entity 1, and entity 2, such as predicted futures 1-N of example 310 as described with respect to FIG. 3B.

[0086] 5B shows an example structural diagram for determining a predicted position at a second time using a first type prediction and a second type prediction. The selected pose (e.g., the object for which the prediction is being performed) and the information on which the selected pose depends (e.g., the information and / or object used to determine the prediction) may be determined using key 502. In some examples, the first type prediction may correspond to the first type prediction as described with respect to FIG. 3A and / or the second type prediction may correspond to the second type prediction as described with respect to FIG. 3B.

[0087] 5A, a predicted position of entity 2 in time horizon 2 may be determined using a first type prediction and a second type prediction. For example, determining a predicted position of entity 2 in time horizon 2 using a first type prediction may be determined independently from information associated with the vehicle and / or entity 1 in any given time horizon. That is, with a first type prediction, a predicted position of entity 2 in time horizon 2 may be based solely on information associated with entity 2 in time horizon 2.

[0088] Alternatively, using the second type of prediction, determining the predicted position of entity 2 in time horizon 2 may be determined relative to information of all entities from all previous time horizons. That is, using the second type of prediction, the predicted position of entity 2 in time horizon 2 may be based on information associated with the vehicle, entity 1, and entity 2 in time horizon 0 and time horizon 1. Given that time horizon 2 depends on time horizon 1, which depends on time horizon 0, the graph structure may maintain edge features associated with other entities in previous time horizons and update the features accordingly. In this manner, the predicted position of entity 2 may be based on the previous position of entity 2 in time horizon 1 and the respective positions of the vehicle and entity 1 in time horizon 1, and one or more predicted futures may be determined for the environment including the vehicle, entity 1, and entity 2, such as predicted futures 1-N of example 310 described with respect to FIG. 3B.

[0089] FIG. 6 illustrates a block diagram of an example architecture 600 for encoding features into a GNN and sampling predicted positions of objects from distribution data output by the GNN. In some examples, the architecture 600 may include a first ML model 602, a second ML model 604, and / or a third ML model 606. In some examples, the first ML model 602 may be configured as an RNN and / or may correspond to the first ML model as described with respect to FIG. 1. Additionally or alternatively, the second ML model 604 may be configured as an RNN, a CNN, and / or may correspond to the second ML model as described with respect to FIG. 1. Additionally or alternatively, the third ML model 606 may be configured as a GNN and / or may correspond to the neural network 142 and / or the GNN as described with respect to FIG. 1.

[0090] The first ML model 602 may be configured to process static scene data 608 associated with the environment. In some examples, the static scene data 608 may be determined based on sensor data captured by a vehicle in the environment. Additionally or alternatively, the static scene data 608 may be represented in a plan, elevation, or side view, although not depicted in FIG. 6 . The first ML model 602 may be configured to process the static scene data 608 to determine scene context features 610 associated with the environment. The scene context features 610 may include multiple channels corresponding to features of the environment at the current time step, where each channel may capture three-dimensional features of the environment represented by the sensor data and represent a feature (or feature vector) at a location of the environment corresponding to the width and / or height of the environment.

[0091] The second ML model 604 may be configured to process an entity history 612 associated with the environment. In some examples, the entity history 612 may be based on previous iterations of the GNN and may include features associated with each entity in the environment at each of the previous time steps. The second ML model 604 may be configured to process the entity history 612 to determine entity features 614 for each entity at the current time step. The entity features 614 may include a number of channels corresponding to the features of each entity in the environment at the current time step.

[0092] Once the scene context features 610 and the entity features 614 have been determined for the current time step, an inference operation 616 may be determined. In some examples, the inference operation 616 may include updating nodes of the third ML model 606 (e.g., a GNN) based on the entity features 614 and the scene context features 610 at the given time step. As a result of the inference operation 616, the third ML model 606 may output a distribution of predicted positions of each of the entities in the environment.

[0093] A sampling technique may be employed to sample a single predicted location from each of the distributions of predicted locations of the entity. The sampling technique 618 may include various sampling techniques, where different sampling techniques 618 may be used to sample from the distributions of predicted locations of the entity between separate time steps and / or to sample from different distributions of predicted locations of the entity at the same time step. In some examples, the sampling technique 618 may be configured as a Gaussian sampling technique (e.g., following a sampling pattern based on a Gaussian distribution), a maximum likelihood sampling technique (e.g., taking an average of the minimum, maximum, or mean value of the distribution, etc.), or an overwrite sampling technique (e.g., selecting a particular location of the object that may or may not be included in the distribution). Additionally or alternatively, the sampling technique 618 for sampling the predicted location distributions may be determined based on a classification type of the graph node (e.g., whether the graph node is associated with an autonomous vehicle, an object, a particular type of object, etc.), a predicted location of the predicted location distribution (e.g., whether the predicted location is within a predefined region), a distribution type of the predicted location distribution, an input associated with the predicted location distribution (e.g., the autonomous vehicle reference trajectory 620), etc.

[0094] Once multiple predicted positions have been determined for each entity corresponding to multiple time steps (e.g., eight predicted positions for each entity corresponding to eight time steps), a predicted future 622 for the environment may be determined. In some examples, the predicted future 622 may include a pose of each entity within the environment over each of the time steps. In some examples, the predicted future 622 may be utilized to determine a predicted trajectory of the entity, such as, for example, predicted futures 1-N as described with respect to FIG. 3B.

[0095] 7 shows a block diagram of an example 700 for updating nodes of a GNN and sampling predicted positions of objects from distribution data output by the GNN over a single time horizon. In some examples, the block diagram illustrated in the example 700 may be configured as part of the example architecture 600 as illustrated in FIG.

[0096] For example, take an environment 702 at a first time horizon (labeled as time horizon 0 in FIG. 7 ) that includes a first object 704, a second object 706, and / or a third object 708. Using a graph structure, such as a graph neural network (GNN), and techniques described herein, a predicted future set associated with the objects can be determined for the environment 702.

[0097] At time horizon 0, a first node 712, a second node 714, and a third node 716 of the GNN 710 may be determined and / or updated. In some examples, the first node 712 associated with the first object 704 may be determined using scene context features 610 corresponding to the position of the first object 704 in the environment 702 and one or more features associated with the first object 704. Additionally or alternatively, the second node 714 associated with the second object 706 may be determined using scene context features 610 corresponding to the position of the second object 706 in the environment 702 and one or more features associated with the second object 706. Additionally or alternatively, the third node 716 associated with the third object 708 may be determined using scene context features 610 corresponding to the position of the third object 708 in the environment 702 and one or more features associated with the third object 708.

[0098] Further, at time horizon 0, the GNN 710 may include a first edge 718 connecting the first node 704 and the second node 706, a second edge 720 connecting the first node 704 and the third node 708, and / or a third edge 722 connecting the second node 706 and the third node 708. In some examples, each of the edges 718, 720, and / or 722 may represent a position of an object associated with the node connected to the edge relative to a position of another object associated with another node connected to the edge. In some examples, the edges 718, 720, and / or 722 may be encoded with various features associated with the objects as discussed herein and configured as bidirectional edges of the GNN 710. That is, with respect to the first node 712, the first edge 718 may represent a position of the second object 706 relative to a position of the first object 704. Additionally or alternatively, with respect to the second node 714 , the first edge 720 may represent the position of the first object 704 relative to the position of the second object 706 .

[0099] After updating the GNN 710 with features associated with the objects 704, 706, and / or 708 in time horizon 0 and the corresponding scene context features 610, the nodes 712, 714, and / or 716 of the GNN 710 may be decoded and a first distribution of predicted positions 724 associated with a first object, a second distribution of predicted positions 726 associated with a second object, and / or a third distribution of predicted positions 728 associated with a third object may be determined for time horizon 1. In some examples, predicted positions of objects in the future (e.g., time horizon 1) may be sampled from the distribution of predicted positions. In some examples, various sampling techniques may be used to sample the distribution of predicted positions, as described with respect to Figures 1 and 10. Additionally or alternatively, the distribution of predicted positions may be evaluated to determine a subset of predicted positions within the distribution that meet a criterion before being sampled, as described with respect to Figures 1 and 11. Additionally or alternatively, the predicted location of an object sampled from the distribution of predicted locations may be overwritten, as described with respect to Figures 1, 2 and 9.

[0100] After decoding the nodes of the GNN 710 to determine a distribution of predicted positions 724, 726, and / or 728 of objects 704, 706, and / or 708 in time horizon 1, the positions of each of the objects 704, 706, and / or 708 may be sampled from the distribution and nodes 712, 714, and / or 716 may be updated with features of the objects 704, 706, and / or 708 in time horizon 1 and corresponding scene context features 610. This process may be repeated for any number of future time horizons. Additionally or alternatively, a predicted trajectory of each of the objects 704, 706, and / or 708 may be determined based on the predicted positions of the objects in each of the time horizons.

[0101] 1, 2, and 8-11 illustrate an exemplary process according to an example of the present disclosure. The process is illustrated as a logical flow graph, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be omitted or combined, in any order and / or in parallel, to implement the process.

[0102] 8A and 8B show an example process 800 for ingesting sensor data, encoding features into nodes and edges of a GNN, and sampling predicted locations of objects from distribution data output by the GNN. In some examples, the sensor data, nodes, edges, GNN, predicted locations, objects, and / or distribution data may correspond to the sensor data 108, nodes 132, 134, edges 136, neural network 142, predicted locations, and / or objects 130, as described with respect to FIG.

[0103] Process 800 begins at operation 802 and involves receiving sensor data from a sensor associated with a vehicle in an environment. In some examples, the environment may include a first object.

[0104] At operation 804, the process 800 may include determining first data representative of a first state of the first object. In some examples, the first data may be determined based at least in part on the sensor data.

[0105] At operation 806, process 800 may include inputting the first data into a first machine learning (ML) model. Additionally or alternatively, process 800 may include receiving second data from the first ML model, the second data including a first feature associated with the first object, based at least in part on inputting the first data into the first ML model. In some examples, the first ML model may be configured as a recurrent neural network (RNN).

[0106] At operation 808, process 800 may include determining third data representing a top-down view of the environment. In some examples, the third data may be determined based at least in part on map data associated with the environment. In some examples, the map data may be received from a map server associated with the vehicle.

[0107] At operation 810, the process 800 may include inputting the third data into a second ML model. Additionally or alternatively, the process 800 may include receiving fourth data including a second feature associated with the environment based at least in part on the input of the third data into the second ML model. In some examples, the second ML model may be different from the first ML model. Additionally or alternatively, the second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0108] At operation 812, the process 800 may include associating the second data and the fourth data with a first node of a graph neural network (GNN) representing the first object.

[0109] In operation 814, process 800 may include determining a second node of the GNN that represents the vehicle.

[0110] At operation 816, the process 800 may include determining a first edge connecting the first node and the second node. In some examples, the first position of the first object is represented relative to the second position of the vehicle at the first edge.

[0111] At operation 818, process 800 may include performing a first inference operation on the GNN. Additionally or alternatively, process 800 may include determining a first output representing first distribution data for the first object based at least in part on the first inference operation. In some examples, the first distribution data may include a first predicted location of the first object. Additionally or alternatively, the process may include determining the first predicted location of the first object based on the first inference operation.

[0112] At operation 820, process 800 may include determining whether a threshold amount of predicted positions for the first object have been determined. For example, a predicted trajectory of the first object may be determined using one or more predicted positions of the first object. As an example, operation 820 may include determining that a threshold amount of predicted positions have been determined to determine the predicted trajectory. As another example, operation 820 may include determining that a threshold amount of predicted positions have not been determined to determine the predicted trajectory. The process may then include operation 818 of restarting a portion of process 800 (at a subsequent time step) based on determining that a threshold amount of predicted positions have not been determined to determine the predicted trajectory.

[0113] At operation 822, process 800 may include determining a first trajectory for the first object. In some examples, determining the first trajectory for the first object may be based at least in part on a first predicted position of the first object.

[0114] In some examples, the vehicle is an autonomous vehicle that traverses the environment. Additionally or alternatively, process 800 may include controlling the autonomous vehicle based at least in part on the first trajectory for the first object.

[0115] Additionally or alternatively, the process 800 may include determining fifth data representing a second state of a second object in the environment based at least in part on the sensor data. Additionally or alternatively, the process 800 may include receiving sixth data including a third feature associated with the second object based at least in part on inputting the fifth data into the second ML model. Additionally or alternatively, the process 800 may include associating the fourth data and the sixth data with a third node of the GNN representing the second object. Additionally or alternatively, the process 800 may include determining a second edge connecting the first node and a third node, where the third position of the second object is represented relative to the first position of the first object. Additionally or alternatively, the process 800 may include determining a third edge connecting the second node and a third node, where the third position of the second object is represented as a relative position to the second position of the vehicle. Additionally or alternatively, process 800 may include performing a second inference operation on the GNN. Additionally or alternatively, process 800 may include determining a second output representing second distribution data for the second object based at least in part on the second inference operation, the second distribution data including a second predicted location for the second object. Additionally or alternatively, process 800 may include determining a second predicted location of the second object based at least in part on sampling from the second distribution data. Additionally or alternatively, process 800 may include determining a first trajectory of the first object and a second trajectory of the second object based at least in part on the first predicted location of the first object and the second predicted location of the second object.

[0116] In some examples, the first location of the first object may be associated with a first time. Additionally or alternatively, the first predicted location of the first object may be associated with a second time subsequent to the first time. Additionally or alternatively, the process 800 may include updating a first node of the GNN representing the first object based at least in part on the fourth data and the first predicted location of the first object, the first predicted location of the first object being represented relative to the second location of the vehicle. Additionally or alternatively, the process 800 may include performing a second inference operation on the GNN. Additionally or alternatively, the process 800 may include determining a second output representing second distribution data of the first object based at least in part on the second inference operation, the second distribution data including the second predicted location of the first object. Additionally or alternatively, process 800 may include determining a second predicted position of the first object at a third time subsequent to the second time based at least in part on sampling from the second distribution data. In some examples, determining the first trajectory of the first object may be based at least in part on the first predicted position of the first object and / or the second predicted position of the first object.

[0117] In some examples, the first data may include a second state of a second object in the environment.

[0118] Additionally or alternatively, process 800 may include determining a second predicted position of the second object based at least in part on sampling from the second distribution data including the second predicted position for the second object. In some examples, determining the first trajectory of the first object may be based at least in part on the first predicted position of the first object and the second predicted position of the second object.

[0119] Additionally or alternatively, process 800 may include determining a first feature associated with the object based at least in part on data representing at least one of an object pose, an object position, an object acceleration, an object velocity, an object size, an object type, and / or an object lighting condition.

[0120] Additionally or alternatively, process 800 may include determining a second characteristic associated with the environment based at least in part on data representing at least one of: additional objects in the environment, types of additional objects in the environment, lighting conditions of additional objects in the environment, and / or lane markings in the environment.

[0121] 9 is an example process 900 for ingesting sensor data, encoding features into nodes and edges of a GNN, determining that a first predicted location of an object sampled from distribution data output by the GNN is outside a specified region, and determining a second predicted location of the object based on map data. In some examples, the sensor data, nodes, GNN, first predicted location, specified region, second predicted location, map data, objects, and / or distribution data may correspond to the sensor data 208, nodes 228, graph structure 226, first predicted location 246, region 240, second predicted location 250, map data, objects 242, and / or distribution data 244 as described with respect to FIG.

[0122] Process 900 begins at operation 902 and involves receiving sensor data from a sensor associated with a vehicle in an environment. In some examples, the environment may include objects.

[0123] At operation 904, the process 900 may include determining a first feature associated with the object. In some examples, the first feature may be determined based at least in part on the sensor data. Additionally or alternatively, the process 900 may include determining a first state of the object, inputting the first state of the object into a first machine learning (ML) model, and / or receiving first feature data associated with the object based at least in part on inputting the first state of the object into the first ML model. In some examples, the first ML model may be configured as a recurrent neural network (RNN).

[0124] At operation 906, the process 900 may include determining a feature map that represents a region associated with the environment. In some examples, the feature map may be determined based at least in part on map data associated with the environment. In some examples, the feature map may correspond to feature map 218 as described with respect to FIG.

[0125] At operation 908, the process 900 may include determining a second feature associated with the environment. In some examples, the second feature may be based at least in part on a current position and feature map of an object in the environment. Additionally or alternatively, the process 900 may include determining a top-down view of the environment, inputting the top-down view into a second ML model, and / or receiving second feature data based at least in part on inputting the top-down view of the environment into the second ML model. In some examples, the second ML model may be different from the first ML model. Additionally or alternatively, the second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0126] At operation 910, the process 900 may include determining nodes of a graph neural network (GNN) that represent the first feature and the second feature.

[0127] At operation 912, the process 900 may include determining a first output representing first distribution data associated with the object. In some examples, the first output may be based at least in part on the GNN. Additionally or alternatively, the first distribution data may include a first predicted position of the object in the future. Additionally or alternatively, the first output may represent the first predicted position of the object in the future.

[0128] At operation 914, process 900 may include determining whether the first predicted location is outside the domain of the environment. As an example, operation 914 may include determining that the first predicted location is inside the domain of the environment. Process 900 may then include operation 916 based on determining that the first predicted location is inside the domain of the environment. As another example, operation 914 may include determining that the first predicted location is outside the domain of the environment. Process 900 may then include operation 918 based on determining that the first predicted location is outside the domain of the environment.

[0129] In operation 916, process 900 may include controlling the vehicle based at least in part on the first predicted position.

[0130] At operation 918, process 900 may include determining a second predicted location of the object. In some examples, the second predicted location of the object may be based at least in part on the first predicted location being outside of the region and / or map data.

[0131] In operation 920, process 900 may include controlling the vehicle based at least in part on the second predicted position.

[0132] Additionally or alternatively, process 900 may include determining a first predicted location based at least in part on sampling from the first distribution data.

[0133] Additionally or alternatively, process 900 may include determining the reference line based at least in part on the current position of the object in the environment and the map data. In some examples, determining the second predicted position of the object may be based at least in part on projecting the first predicted position onto the reference line.

[0134] Additionally or alternatively, the process 900 may include determining that a second predicted location of the object in the environment is outside a region associated with the feature map. Additionally or alternatively, the process 900 may include determining a third feature associated with the environment based at least in part on the second predicted location of the object being outside the region, the third feature being a null feature. Additionally or alternatively, the process 900 may include updating a node of the GNN based at least in part on the third feature. Additionally or alternatively, the process 900 may include determining a second output representing second distribution data associated with the object based at least in part on the GNN, the second distribution data including the second predicted location of the object. Additionally or alternatively, the process 900 may include determining that a third predicted location of the object is outside a region of the environment based at least in part on the set of second distributions. Additionally or alternatively, the process 900 may include determining a fourth predicted location of the object based at least in part on the map data. Additionally or alternatively, process 900 may include controlling the vehicle based at least in part on the second predicted position of the object and the fourth predicted position of the object.

[0135] In some examples, a first distance between the current position of the object in the environment and the first predicted position of the object may be the same as a second distance between the current position of the object in the environment and the second predicted position of the object.

[0136] In some examples, determining the second predicted position of the object may be based at least in part on a vehicle dynamics model.

[0137] In some examples, the region may be a bounded region, and a portion of the feature map that corresponds to the bounded region may be used to plan a trajectory of the vehicle through the environment.

[0138] 10 is an example process 1000 for ingesting sensor data, encoding features into nodes and edges of a GNN, determining a sampling technique for sampling distribution data output by the GNN, and controlling a vehicle based on a predicted location of an object sampled from the distribution data using the sampling technique. In some examples, the sensor data, nodes, edges, GNN, predicted location, object, and / or distribution data may correspond to the sensor data 108, nodes 132, 134, edges 136, neural network 142, predicted location, object 130, and / or predicted location distribution 146, as described with respect to FIG.

[0139] The process 1000 begins at operation 1002 and involves receiving sensor data from a sensor associated with a vehicle in an environment. In some examples, the environment may include an object.

[0140] At operation 1004, the process 1000 may include determining a first feature associated with one of the object or the vehicle. In some examples, the first feature may be based at least in part on the sensor data. Additionally or alternatively, the process 1000 may include determining a first state of the object or the vehicle, inputting the first state of the object or the vehicle into a first machine learning (ML) model, and / or receiving the first feature associated with the object or the vehicle based at least in part on inputting the first state of the object or the vehicle into the first ML model. In some examples, the first ML model may be configured as a recurrent neural network (RNN).

[0141] At operation 1006, the process 1000 may include determining a second feature associated with the environment. In some examples, the second feature associated with the environment may be based at least in part on map data associated with the environment. Additionally or alternatively, the process 1000 may include determining a top-down view of the environment, inputting the top-down view into a second ML model, and / or receiving the second feature data based at least in part on inputting the top-down view of the environment into the second ML model. In some examples, the second ML model may be different from the first ML model. Additionally or alternatively, the second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0142] At operation 1008, the process 1000 may include determining a graph node of a graph neural network (GNN). In some examples, the graph node may be based on the first feature and the second feature.

[0143] At operation 1010, the process 1000 may include determining a first output representing distribution data associated with the node. In some examples, the first output may be based at least in part on the inference of the GNN. Additionally or alternatively, the distribution data includes a predicted position of an object or vehicle in the environment.

[0144] At operation 1012, the process 1000 may include determining a sampling technique for sampling the distribution data. In some examples, the sampling technique may be based at least in part on a classification type of the graph node, a predicted position of the distribution data, a distribution type of the distribution data, and / or an input associated with the distribution data.

[0145] In operation 1014, the process 1000 may include determining a first predicted position of the object or vehicle based at least in part on the sampling technique.

[0146] At operation 1016, process 1000 may include determining whether a threshold amount of predicted positions have been sampled from the distribution data. For example, a predicted trajectory of an object may be determined using one or more predicted positions of the object. As an example, operation 1016 may include determining that a threshold amount of predicted positions have been sampled from the distribution data to determine the predicted trajectory. As another example, operation 1016 may include determining that a threshold amount of predicted positions have not been sampled from the distribution data to determine the predicted trajectory. The process may then include operation 1012 of restarting a portion of process 1000 (at a subsequent time step) based on determining that a threshold amount of predicted positions have not been sampled from the distribution data to determine the predicted trajectory.

[0147] In operation 1018, the process 1000 may include controlling the vehicle based at least in part on the first predicted position of the object.

[0148] Additionally or alternatively, process 1000 may include determining that a classification type of the graph node indicates that the graph node is associated with an object, where the sampling technique includes a first sampling technique. Additionally or alternatively, process 1000 may include determining that a classification type of the graph node indicates that the graph node is associated with a vehicle, where the sampling technique may include a second sampling technique different from the first sampling technique.

[0149] Additionally or alternatively, the process 1000 may include determining a boundary region of the environment used to plan a trajectory of the vehicle. Additionally or alternatively, the process 1000 may include determining that a predicted location of the distribution data is outside of the boundary region based at least in part on the distribution data. Additionally or alternatively, the process 1000 may include determining a sampling technique based at least in part on the predicted location of the distribution data being outside of the boundary region. Additionally or alternatively, the process 1000 may include determining a first predicted location of the object based at least in part on the sampling technique and map data associated with the environment.

[0150] Additionally or alternatively, the process 1000 may include determining that a distribution type of the distribution data is a first distribution type. Additionally or alternatively, the process 1000 may include determining a first predicted location of the object or vehicle based at least in part on sampling the distribution data using a sampling technique, where the sampling technique includes a Gaussian sampling technique. Additionally or alternatively, the process 1000 may include determining that a distribution type of the distribution data is a second distribution type. Additionally or alternatively, the process 1000 may include determining a first predicted location of the object or vehicle based at least in part on sampling the distribution data using a sampling technique. In some examples, the first predicted location may be an average of the predicted locations of the distribution data.

[0151] In some examples, the vehicle may be an autonomous vehicle. Additionally or alternatively, process 1000 may include receiving input data from a planning component of the autonomous vehicle, where the first predicted position of the object or vehicle may be based at least in part on the input data. In some examples, the input data may include a reference trajectory of the autonomous vehicle through the environment.

[0152] Additionally or alternatively, process 1000 may include receiving input data, where the input data may indicate a selected position of the object or vehicle in an environment that differs from the predicted position. Additionally or alternatively, process 1000 may include determining a predicted trajectory of the object or vehicle based on the selected position.

[0153] In some examples, the sampling technique may be the first sampling technique, and / or the first predicted location may be associated with the first time. Additionally or alternatively, the process 1000 may include updating a graph node having a third feature associated with the environment based at least in part on the first predicted location of the object or vehicle. Additionally or alternatively, the process 1000 may include determining a second output representing distribution data associated with the node based at least in part on the graph structure, the distribution data including a predicted location of the object or vehicle in the environment at a second time after the first time. Additionally or alternatively, the process 1000 may include determining a second sampling technique for sampling the distribution data, the second sampling technique may be different from the first sampling technique. Additionally or alternatively, the process 1000 may include determining a second predicted location of the object or vehicle in the environment at the second time based at least in part on the second sampling technique. Additionally or alternatively, process 1000 may include determining a predicted trajectory based at least in part on the first predicted position and the second predicted position.

[0154] In some examples, the node may be a first node associated with the object, the distribution data may be first distribution data associated with the first node at a first time, and the sampling technique may be a first sampling technique. Additionally or alternatively, the process 1000 may include associating a third feature associated with the vehicle and a second feature associated with the environment with a second node of the graph structure. Additionally or alternatively, the process 1000 may include determining a second output representing second distribution data associated with the second node based at least in part on the graph structure, the second distribution data including a second predicted position of the vehicle within the environment at the first time. Additionally or alternatively, the process 1000 may include determining a first sampling technique for sampling the first distribution data based at least in part on a classification type of the first node. Additionally or alternatively, the process 1000 may include determining a second sampling technique for sampling the second distribution data based at least in part on the first classification type of the second node, where the second sampling technique may differ from the first sampling technique. Additionally or alternatively, the process 1000 may include determining a second predicted position of the vehicle at the first time based at least in part on the second sampling technique. Additionally or alternatively, the process 1000 may include determining a predicted trajectory of the object or vehicle based at least in part on the first predicted position or the second predicted position.

[0155] 11 is an example process 1100 for capturing sensor data, encoding features into nodes and edges of a GNN, evaluating first distribution data output by the GNN to determine second distribution data including a subset of predicted locations included in the first distribution data that satisfy a criterion, and controlling a vehicle based on predicted locations of objects sampled from the second distribution data. In some examples, the sensor data, nodes, edges, GNN, predicted locations, objects, criteria, first distribution data, and / or second distribution data may correspond to the sensor data 108, nodes 132, 134, edges 136, neural network 142, predicted locations, objects 130, criteria, distribution data, and / or subsets of distribution data, as described with respect to FIG.

[0156] Process 1100 begins at operation 1102 and includes receiving sensor data from a sensor associated with a vehicle in an environment at a first time. In some examples, the environment may include an object.

[0157] At operation 1104, process 1100 may include determining a first feature associated with the object. In some examples, the first feature may be based at least in part on the sensor data. Additionally or alternatively, process 1100 may include determining a first state of the object, inputting the first state of the object into a first machine learning (ML) model, and / or receiving first feature data associated with the object based at least in part on inputting the first state of the object into the first ML model. In some examples, the first ML model may be configured as a recurrent neural network (RNN).

[0158] At operation 1106, process 1100 may include determining a second feature associated with the environment. In some examples, the second feature may be based at least in part on the sensor data. Additionally or alternatively, process 1100 may include determining a top-down view of the environment, inputting the top-down view into a second ML model, and / or receiving second feature data based at least in part on inputting the top-down view of the environment into the second ML model. In some examples, the second ML model may be different from the first ML model. Additionally or alternatively, the second ML model may be configured as an RNN or a convolutional neural network (CNN).

[0159] In operation 1108, the process 1100 may include associating a first feature associated with the object and a second feature associated with the environment to a node of a graph neural network (GNN).

[0160] At operation 1110, the process 1100 may include determining a first output representing first distribution data including predicted locations where the object will be in the environment at a second time after the first time. In some examples, the first output may be determined based at least in part on the GNN and / or a criterion. In some examples, the first distribution data may include only predicted locations that meet the criterion.

[0161] At operation 1112, the process 1100 may include determining a first predicted location of the object. In some examples, the first predicted location of the object may be based at least in part on sampling from the second distribution data.

[0162] At operation 1114, process 1100 may include determining whether a threshold amount of predicted positions have been determined. For example, a predicted position of the first object may be determined based on sampling from the distribution data, and the one or more predicted positions may be used to determine a trajectory of the first object. As an example, operation 1114 may include determining that a threshold amount of predicted positions have been determined. As another example, operation 1114 may include determining that a threshold amount of predicted positions have not been determined. Process 1100 may then include operation 1110 of restarting a portion of process 1100 (at a subsequent time step) based on determining that a threshold amount of predicted positions are not included in the second distribution data.

[0163] In operation 1116, the process 1100 may include controlling the vehicle based on the first predicted position of the object.

[0164] In some examples, the sensor is associated with an autonomous vehicle traversing the environment. Additionally or alternatively, process 1100 may include receiving a trajectory of the vehicle through the environment from a planning component associated with the autonomous vehicle. Additionally or alternatively, process 1100 may include determining the criterion based at least in part on the trajectory of the autonomous vehicle.

[0165] Additionally or alternatively, process 1100 may include determining an intent associated with the object based at least in part on a first characteristic associated with the object and a second characteristic associated with the environment. Additionally or alternatively, process 1100 may include determining the criterion based at least in part on an intent associated with the object.

[0166] Additionally or alternatively, the process 1100 may include receiving map data associated with the environment. Additionally or alternatively, the process 1100 may include determining rules associated with the environment based at least in part on the map data and the sensor data. In some examples, the rules may include at least one of stop signs, street lights, zones in the environment, and / or speed limits. In some examples, the zones in the environment may be defined zones (e.g., school zones at certain times of the day, long-term highway construction zones, etc.) and / or undefined zones (e.g., construction zones that are set or lifted at unspecified times that may be indicated by signs, the presence of traffic control workers, etc.). Additionally or alternatively, the process 1100 may include determining criteria based at least in part on the rules associated with the environment.

[0167] In some examples, the criteria may be based at least in part on object characteristics associated with the object, hi some examples, the object characteristics include at least one of a maximum turning radius associated with the object, a maximum speed associated with the object, a maximum acceleration associated with the object, a lane change maneuver associated with the object, a turning maneuver associated with the object, and / or an hostile behavior maneuver associated with the object.

[0168] Additionally or alternatively, process 1100 may include determining the criterion based at least in part on a first characteristic of the object. In some examples, the first characteristic includes at least one of a current pose of the object, a current position of the object, a current acceleration of the object, a current velocity of the object, a size of the object, a type of the object, and / or a lighting condition of the object.

[0169] In some examples, the object may be a first object, the predicted location may be a first predicted location, and the criterion may be a first criterion. Additionally or alternatively, the process 1100 may include determining a third feature associated with a second object in the environment based at least in part on the sensor data. Additionally or alternatively, the process 1100 may include associating the third feature associated with the second object and the second feature associated with the environment to a second node of the graph structure. Additionally or alternatively, the process 1100 may include determining a second output representing third distribution data of the second object based at least in part on the graph structure, the third distribution data including a second predicted location of the second object in the future. Additionally or alternatively, the process 1100 may include determining a second criterion based at least in part on the first predicted location of the first object. Additionally or alternatively, process 1100 may include evaluating the third distribution data to determine which of the second predicted positions in the third distribution data meet a second criterion. Additionally or alternatively, process 1100 may include determining fourth distribution data for the second object based at least in part on evaluating the third distribution data for the second object, the fourth distribution data including a second subset of the second predicted positions for the second object. Additionally or alternatively, process 1100 may include determining a second predicted position of the second object based at least in part on sampling from the set of fourth distributions. Additionally or alternatively, process 1100 may include determining a second predicted trajectory of the second object based at least in part on the second predicted position of the second object and the first predicted trajectory of the first object.

[0170] In some examples, the vehicle is an autonomous vehicle traversing the environment. Additionally or alternatively, process 1100 may include determining a third predicted trajectory of the autonomous vehicle based at least in part on the first predicted trajectory of the first object and the second predicted trajectory of the second object. Additionally or alternatively, process 1100 may include controlling the autonomous vehicle based at least in part on the third predicted trajectory.

[0171] 12 is a block diagram of an example system 1200 for implementing the techniques described herein. In at least one example, the system 1200 may include a vehicle 1202, such as the vehicle 106.

[0172] The vehicle 1202 may include a vehicle computing device 1204, one or more sensor systems 1206, one or more emitters 1208, one or more communication connections 1210, at least one direct connection 1212, and one or more drive systems 1214.

[0173] The vehicle computing device 1204 may include one or more processors 1216 and a memory 1218 communicatively coupled to the one or more processors 1216. In the illustrated example, the vehicle 1202 is an autonomous vehicle, however, the vehicle 1202 may be any other type of vehicle, such as a semi-autonomous vehicle, or any other system having at least an image capture device (e.g., a camera-enabled smartphone). In the illustrated example, the memory 1218 of the vehicle computing device 1204 stores a localization component 1220, a perception component 1222, a planning component 1224, one or more system controllers 1226, one or more maps 1228, log data 1230, and a prediction component 1252, etc. 12 as residing in memory 1218 for illustrative purposes, the localization component 1220, the perception component 1222, the planning component 1224, the system controller 1226, the map 1228, and the prediction component 1252 may additionally or alternatively be accessible to the vehicle 1202 (e.g., it is contemplated that the localization component 1220, the perception component 1222, the planning component 1224, the system controller 1226, and the prediction component 1252 may collectively comprise a vehicle controller, as described herein.

[0174] In at least one example, the localization component 1220 may include functionality to receive data from the sensor system 1206 to determine a position and / or orientation (e.g., one or more of an x-position, a y-position, a z-position, a roll, a pitch, or a yaw) of the vehicle 1202. For example, the localization component 1220 may include and / or request / receive a map of the environment and continually determine a position and / or orientation of the autonomous vehicle within the map. In some implementations, the localization component 1220 may utilize SLAM (Simultaneous Self-Localization and Mapping), CLAMS (Calibration, Simultaneous Localization and Mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. to receive image data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, etc., to accurately determine a position of the autonomous vehicle. In some implementations, the localization component 1220 may provide data to various components of the vehicle 1202 to determine an initial position of the autonomous vehicle for determining a path polygon associated with a vehicle path, as discussed herein.

[0175] In some examples, the perception component 1222 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 1222 may provide processed sensor data indicating the presence of an object (e.g., an entity) proximate to the vehicle 1202 and / or the classification of the object as a type of object (e.g., a car, a pedestrian, a bicycle, an animal, a building, a tree, a road surface, a curb, a sidewalk, unknown, etc.). In some examples, the perception component 1222 may provide processed sensor data indicating the presence of a stationary entity proximate to the vehicle 1202 and / or the classification of the object as a type of stationary entity (e.g., a building, a tree, a road surface, a curb, a sidewalk, unknown, etc.).

[0176] In additional or alternative examples, the perception component 1222 may provide processed sensor data indicative of one or more characteristics associated with a detected object (e.g., a tracked object) and / or an environment in which the object is located. In some examples, the characteristics associated with the object may include, but are not limited to, an x-position (global and / or local position), a y-position (global and / or local position), a z-position (global and / or local position), an orientation (e.g., roll, pitch, yaw), a type (e.g., classification) of the object, a velocity of the object, an acceleration of the object, an extent (size) of the object, etc. Characteristics associated with the environment may include, but are not limited to, a presence of another object in the environment, a state of another object in the environment, a time of day, a day of the week, a season, weather, an indication of darkness / lightness, etc.

[0177] The prediction component 1252 may generate one or more probability maps representing predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 1252 may generate one or more probability maps for vehicles, pedestrians, animals, etc., within a threshold distance from the vehicle 1202. In some examples, the prediction component 1252 may measure tracking of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories of the objects based on the observed and predicted behavior. In some examples, the one or more probability maps may represent the intent of one or more objects in the environment.

[0178] In general, the planning component 1224 may determine a path for the vehicle 1202 to follow to traverse an environment. For example, the planning component 1224 may determine various paths and trajectories as well as various levels of detail. For example, the planning component 1224 may determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the path may include a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 1224 may generate instructions for guiding the vehicle 1202 along at least a portion of the path from the first location to the second location. In at least one example, the planning component 1224 may determine how to guide the vehicle 1202 from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may be a trajectory, or a portion of a trajectory. In some examples, multiple trajectories may be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon approach, and one of the multiple trajectories may be selected for vehicle 1202 to navigate.

[0179] In some examples, the planning component 1224 can be communicatively coupled to a prediction component 1252 to generate predicted trajectories for objects in the environment. For example, the prediction component 1252 can generate one or more predicted trajectories for objects within a threshold distance from the vehicle 1202. In some examples, the prediction component 1252 can measure the tracking of the objects and generate trajectories for the objects based on the observed and predicted behavior.

[0180] In at least one example, vehicle computing device 1204 may include one or more system controllers 1226, which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 1202. System controller 1226 may communicate with and / or control corresponding systems of drive system 1214 and / or other components of vehicle 1202.

[0181] The memory 1218 may further include one or more maps 1228 that may be used by the vehicle 1202 to navigate within the environment. For purposes of this discussion, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as topology (such as intersections), roads, mountain ranges, roads, terrain, and the environment in general. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDRA information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, the vehicle 1202 may be controlled based at least in part on the map 1228. That is, map 1228 may be used in conjunction with orientation component 1220, perception component 1222, and / or planning component 1224 to determine a position of vehicle 1202, detect objects in an environment, and / or generate a path and / or trajectory for moving through an environment. Additionally, in some examples, map 1228 may be used in conjunction with a tracking component to determine a position and / or orientation of the vehicle relative to a planned trajectory, such as based on steering angle, speed, acceleration, driving direction, driving gear, and / or gravitational acceleration.

[0182] In some examples, one or more maps 1228 may be stored on a computing device (such as computing device 1234) accessible via network 1236. In some examples, multiple maps 1228 may be stored, for example, based on a characteristic (e.g., type of entity, time of day, day of the week, season, etc.). Storing multiple maps 1228 may have similar memory requirements and may increase the speed at which data in the maps may be accessed.

[0183] 7, memory 1218 can store logged data 1230. Logged data 1230 can represent data input and / or output by each of localization component 1220, perception component 1222, planning component 1224, controller 1226, and / or outputs of their various subcomponents. In at least one example, logged data 1230 can include sensor data captured by sensor system 1206 and provided to one or more of the components of vehicle computing device 1204.

[0184] As can be appreciated, the components described herein (e.g., the localization component 1220, the perception component 1222, the planning component 1224, the one or more system controllers 1226, and the one or more maps 1228) are described as separated for purposes of illustration. However, the operations performed by the various components may be combined or performed in any other component. For example, in an illustrative example, the functionality of the tracking component as described above may be performed by the planning component 1224. However, in other examples, the tracker component may include a separate component that is separate from the planning component 1224.

[0185] In some examples, some or all aspects of the components discussed herein may include any models, techniques, and / or machine learning techniques. For example, in some examples, the components in memory 1218 (and memory 1232, described below) may be implemented as neural networks.

[0186] In at least one example, the sensor system 1206 may include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 1206 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors positioned at corners, front, back, sides, and / or top of the vehicle 1202. As another example, the camera sensors may include multiple cameras positioned at various locations about the exterior and / or interior of the vehicle 1202. The sensor system 1206 may provide input to the vehicle computing device 1204. Additionally or alternatively, the sensor system 1206 may transmit sensor data at a particular frequency, after a predetermined period of time, or in near real-time via one or more networks 1236 to one or more computing devices 1234.

[0187] The vehicle 1202 may also include one or more emitters 1208 for emitting light and / or sound, as described above. The emitters 1208 in this example include internal audio and video emitters for communicating with passengers of the vehicle 1202. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), etc. The emitters 1208 in this example also include external emitters. By way of example and not limitation, the external emitters in this example include lights or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.) to indicate direction of travel, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which emitters include acoustic beam steering technology.

[0188] Vehicle 1202 may also include one or more communication connections 1210 that enable communication between vehicle 1202 and one or more other local or remote computing devices. For example, communication connection 1210 may facilitate communication with other local computing devices on vehicle 1202 and / or drive system 1214. Communication connection 1210 may also enable the vehicle to communicate with other nearby computing devices (e.g., computing device 1234, other nearby vehicles, etc.) and / or one or more remote sensor systems to receive sensor data.

[0189] The communications connection 1210 may include physical and / or logical interfaces for connecting the vehicle computing device 1204 to another computing device or network, such as network 1236. For example, the communications connection 1210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0190] In at least one example, the vehicle 1202 may include one or more drive systems 1214. In some examples, the vehicle 1202 may have a single drive system 1214. In at least one example, when the vehicle 1202 has multiple drive systems 1214, the individual drive systems 1214 may be located at opposite ends of the vehicle 1202 (e.g., at the front and the rear, etc.). In at least one example, the drive system 1214 may include one or more sensor systems for detecting conditions surrounding the drive system 1214 and / or the vehicle 1202. By way of example and not limitation, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring orientation and acceleration associated with the drive system, cameras or other imaging sensors, ultrasonic sensors for acoustically imaging objects surrounding the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be specific to the drive system 1214. In some cases, sensor systems on the drive system 1214 may overlap or supplement corresponding systems of the vehicle 1202 (e.g., sensor system 1206).

[0191] The drive system 1214 may include a high-voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights to illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, and other electrical components). Additionally, the drive system 1214 may include a drive system controller that receives and pre-processes data from the sensor systems and controls the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules for performing various functions of the drive system 1214. Additionally, drive system 1214 may include one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

[0192] In at least one example, the direct connection 1212 may provide a physical interface for coupling one or more drive systems 1214 with the body of the vehicle 1202. For example, the direct connection 1212 may enable the transfer of energy, fluid, air, data, etc. between the drive system 1214 and the vehicle 1202. In some examples, the direct connection 1212 may further releasably secure the drive system 1214 to the body of the vehicle 1202.

[0193] In at least one example, the localization component 1220, the perception component 1222, the planning component 1224, and / or the one or more system controllers 1226, and / or various components thereof may process the sensor data as described above and transmit their respective outputs as logged data 1230 to the computing device 1234 over one or more networks 1236. In at least one example, the vehicle computing device 1204 may transmit the logged data 1230 to the computing device 1234 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0194] In some examples, vehicle 1202 may transmit sensor data to computing device 1234 over network 1236. In some examples, vehicle 1202 may receive sensor data from computing device 1234 over network 1236. The sensor data may include raw sensor data and / or processed sensor data and / or representations of sensor data. In some examples, the sensor data (raw or processed) may be transmitted and / or received as one or more log files.

[0195] The computing device 1234 may include a processor 1238 and a memory 1232 that stores an object feature component 1240 , an environmental feature component 1242 , a distribution component 1244 , a sampling component 1246 , and / or a modeling component 1248 .

[0196] The object features component 1240 may be configured to determine features associated with objects in the environment based on sensor data representative of the environment. In some examples, the object features component 1240 may be configured to determine data representative of a state of an object in the environment. In some examples, the state may include, but is not limited to, an object pose, an object position, an object acceleration, an object velocity, an object size, an object type, an object lighting condition, and the like. Additionally or alternatively, the object features component 1240 may be configured to determine features associated with the object by processing data representative of the object state with a machine learning (ML) model. For example, the data representative of the object state may be input into the ML model and processed to determine output data including the features associated with the object. In some examples, the ML model may be configured as a recurrent neural network (RNN). In some examples, the output data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the object. For example, the features may be configured as a feature vector including vectorized object elements encoded as a feature vector.

[0197] The environment features component 1242 may be configured to determine features associated with the environment that correspond to the location of objects within the environment. In some examples, the environment features component 1242 may be configured to determine data representing a field of view of the environment. In some examples, the data may be represented in a plan view or a top-down view. In other examples, the data may be represented in an elevation view or a side view. The data representing the field of view of the environment may be determined using sensor data representing the environment and / or map data associated with the environment and received from a map data server. In some examples, the field of view of the environment may represent various elements within the environment, including, but not limited to, roads, lanes, curbs, shoulders, crosswalks, buildings, trees, medians, road signs, traffic signals, speed limits, etc. Additionally or alternatively, the environment features component 1242 may determine features associated with the environment by processing the data representing the field of view of the environment with an ML model. For example, the data representing the field of view of the environment may be input into the ML model and processed to determine output data including the features associated with the environment. The ML model may be configured as an RNN or a convolutional neural network (CNN). In some examples, the output data may be configured as multi-dimensional data and may include any number of channels corresponding to the number of features associated with the environment at the location of the object within the environment, e.g., the features may be configured as feature vectors that include vectorized environment elements encoded as feature vectors.

[0198] The distribution component 1244 may be configured to decode node and / or edge features connecting nodes of the GNN to determine a distribution of predicted locations of objects in the environment over a future time horizon. Additionally or alternatively, the distribution component 1244 may be configured to evaluate the distribution of predicted locations to determine a subset of predicted locations that meet a criterion to allow for more efficient sampling of predicted locations (e.g., eliminating impossible real-world scenarios, influencing objects to behave in a particular manner, etc.). In some examples, the criterion may be based on a type of object, a characteristic of the object, map data, input data, etc.

[0199] The sampling component 1246 may be configured to use various sampling techniques to sample the distribution of predicted positions to determine predicted positions of objects in the environment at future time horizons. The various sampling techniques include, but are not limited to, Gaussian sampling (e.g., following a sampling pattern based on a Gaussian distribution), maximum likelihood sampling (e.g., taking an average of the minimum, maximum, or mean of the distribution), or overwrite sampling (e.g., selecting a specific position of the object that may or may not be included in the distribution). The sampling component 1246 may be configured to determine the sampling technique based on the classification type of the graph node (e.g., whether the graph node is associated with an autonomous vehicle, an object, a particular type of object, etc.), the predicted position of the predicted position distribution (e.g., whether the predicted position is within a predefined region), the distribution type of the predicted position distribution, an input associated with the predicted position distribution (e.g., a reference trajectory associated with the object), etc. The sampling component 1246 may be configured to employ different sampling techniques to sample distributions of two different objects at the same time horizon and / or to sample distributions of one object at different time horizons.

[0200] The modeling component 1248 may be configured to determine a graph structure and / or a graph neural network (GNN) including nodes representing objects in the environment and edges connecting the nodes. In some examples, the modeling component may be configured to update the GNN and / or perform instance operations. Additionally or alternatively, the modeling component 1248 may be configured to encode features associated with the objects and / or features associated with the environment at the positions of the objects in the environment into the nodes of the GNN. Additionally or alternatively, the modeling component 1248 may be configured to encode relative features of the objects into the edges of the GNN.

[0201] In various examples, the computing device 1234 may include one or more input / output (I / O) devices, such as via one or more interfaces 1250. The interfaces 1250 may include an I / O interface and / or a network interface. The I / O interfaces may include speakers, microphones, cameras, and various user controls (e.g., buttons, joysticks, keyboards, keypads, etc.), tactile output devices, and the like. The network interfaces may include one or more interfaces and hardware components for enabling communication with various other devices over a network or directly. For example, the network interfaces may enable communication over one or more of the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi), and a wired network, as well as short-range communication such as Bluetooth®, Bluetooth® low energy, and the like, as additionally listed elsewhere herein.

[0202] In some examples, a user may view a user interface associated with the sampling component 1246, such as inputting data and / or viewing results via one or more interfaces 1250. In such examples, the interface 1250 may include one or more displays. Depending on the type of computing device, such as a user computing device, a server computing device, etc., the display may employ any suitable display technology. For example, the display may be a liquid crystal display, a plasma display, a light emitting diode display, an OLED (organic light emitting diode) display, an electronic paper display, or any other suitable type of display capable of presenting digital content thereon. In some examples, the display may have a touch sensor associated with the display to provide a touch screen display configured to receive touch input to enable interaction with a graphical user interface presented on the display. Thus, examples herein are not limited to any particular display technology.

[0203] The processor 1216 of the vehicle 1202 and the processor 1238 of the computing device 1234 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 1216 and 1238 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into registers and / or other electronic data that may be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement the encoded instructions.

[0204] The memories 1218 and 1232 are examples of non-transitory computer-readable media. The memories 1218 and 1232 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memories may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, but those shown in the accompanying figures are merely examples relevant to the discussion herein.

[0205] In some examples, some or all aspects of the components discussed herein may include any models, algorithms, and / or machine learning algorithms. For example, in some examples, the components in memories 1218 and 1232 may be implemented as neural networks.

[0206] As described herein, an exemplary neural network is an algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may constitute another neural network, or may constitute any number of layers (whether convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0207] Although discussed in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning or machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification trees (CART), iterative bisection 3 (ID3), chi-squared automated interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, etc.), and so on. The algorithms may include, but are not limited to, deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blend), gradient boosting machine (GBM), gradient boosting regression tree (GBRT), random forest), support vector machine (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0208] Additional example architectures include neural networks such as ResNet50, ResNet52, ResNet101, VGG, DenseNet, and PointNet.

[0209] (Example section) A. A system having one or more processors, the system, when executed by the one or more processors, includes: receiving sensor data from sensors associated with a vehicle in an environment, the environment including a first object; determining first data representing a first state of the first object at a first time based at least in part on the sensor data; receiving second data from a first machine learning (ML) model based at least in part on inputting the first data into a first ML model, the second data including a first feature associated with the first object; determining third data representing a top-down view of the environment based at least in part on map data associated with the environment; and receiving fourth data from the second ML model based at least in part on inputting the third data into the second ML model. receiving the second data, the fourth data including a second feature associated with the environment, associating the second data and the fourth data with a first node of a graph neural network (GNN) representing the first object, determining a second node of the GNN representing the vehicle, and determining a first edge connecting the first node and the second node, wherein a first state of the first object is represented relative to the second state of the vehicle; performing a first inference operation on the GNN; determining a first output representing a first predicted position of the first object at a second time after the first time based at least in part on the first inference operation; and determining a first trajectory of the first object based at least in part on the first predicted position of the first object.

[0210] B. The system of paragraph A, wherein the vehicle is an autonomous vehicle that traverses the environment, and the operation further includes controlling the autonomous vehicle based at least in part on the first trajectory for the first object.

[0211] C. The operations include determining fifth data representing a third state of a second object in the environment based at least in part on the sensor data; receiving sixth data including a third feature associated with the second object based at least in part on inputting the fifth data into the second ML model; associating the fourth data and the sixth data with a third node of the GNN representing the second object; and determining a second edge connecting the first node and the third node, where the third state of the second object is represented with respect to the first state of the first object; and determining a second edge connecting the second node to the third node. determining a third edge connecting the first node to a third node, where a third state of the second object is represented relative to a second state of the vehicle; performing a second inference operation on the GNN; determining a second output representing a second predicted position of the second object at a second time based at least in part on the second inference operation; and determining a first trajectory of the first object and a second trajectory of the second object based at least in part on the first predicted position of the first object and the second predicted position of the second object.

[0212] D. The system of any one of paragraphs A to C, wherein the operations further include updating a first node of the GNN representing the first object based at least in part on the fourth data and the first predicted position of the first object, where the first predicted position of the first object is represented relative to a second state of the vehicle; performing a second inference operation on the GNN; and determining a second output representing a second predicted position of the first object at a third time subsequent to the second time based at least in part on the second inference operation; and determining a first trajectory of the first object based at least in part on the first predicted position of the first object and the second predicted position of the first object.

[0213] E. The system of any one of paragraphs A to D, wherein the first data includes a third state of a second object in the environment, and the operation further includes determining a second predicted position of the second object based at least in part on the GNN, and determining the first trajectory of the first object is based at least in part on the first predicted position of the first object and the second predicted position of the second object.

[0214] F. A method comprising: receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object; determining a first feature associated with the object at a first time based at least in part on the sensor data; determining a second feature associated with the environment based at least in part on sensor data or map data associated with the environment; determining a first node of a graph neural network (GNN) representing the first feature associated with the object and the second feature associated with the environment; determining a second node of the GNN representing the vehicle, wherein an edge connecting the first node and the second node represents a state of at least one of the vehicle or the object as a relative state; determining a first output based at least in part on the GNN representing a first predicted position of the object at a second time after the first time; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object.

[0215] G. The method of claim F, wherein the vehicle is an autonomous vehicle that traverses the environment, and the method further includes controlling the autonomous vehicle based at least in part on the trajectory of the object.

[0216] H. The method of claim F or G, further comprising: determining a second output representing a second predicted position of the vehicle at a second time based at least in part on the GNN; receiving a reference trajectory of the vehicle through the environment from a planning component associated with the vehicle; updating a second node of the GNN based at least in part on the reference trajectory, wherein an edge connecting the first node and the second node represents a candidate position of the vehicle; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object and the candidate position of the vehicle.

[0217] I. The method of any one of clauses F to H, wherein an edge connecting the first node and the second node represents a state of the object relative to the vehicle at a first time, the method further including: determining a feature map associated with the environment based at least in part on the map data; determining a third feature associated with the environment based at least in part on a first predicted position of the object in the environment and the feature map; updating the first node of the GNN based at least in part on the third feature; and determining a second output representing a second predicted position of the object at a third time after the second time based at least in part on the GNN, wherein determining the first predicted trajectory of the object is based at least in part on the first predicted position of the object and the second predicted position of the object.

[0218] J. The method of any one of clauses F-I, further comprising determining a first feature associated with the object based at least in part on data representing at least one of an object pose, an object position, an object acceleration, an object velocity, an object size, an object type, or an object lighting condition.

[0219] K. The method of claim J, further comprising determining a first feature associated with the object based at least in part on inputting the data into a recurrent neural network (RNN).

[0220] L. The method of any one of paragraphs F-K, further comprising determining a second feature associated with the environment based at least in part on data representing at least one of: static objects in the environment, dynamic objects in the environment, traffic signs in the environment, traffic signals in the environment, types of static or dynamic objects in the environment, lighting conditions for static or dynamic objects in the environment, or lane markings in the environment.

[0221] M. The method of any one of paragraphs F to L, wherein the first feature is determined based at least in part on inputting data associated with the object into a first neural network, the data being based at least in part on sensor data, and the second feature is determined at least in part on inputting a field of view of the environment into a second neural network different from the first neural network, the field of view of the environment being based at least in part on the sensor data, and the second feature is associated with a portion of the environment corresponding to the object.

[0222] N. One or more non-transitory computer readable media storing instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations including receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object; determining a first feature associated with the object at a first time based at least in part on the sensor data; determining a second feature associated with the environment based at least in part on the sensor data or map data associated with the environment; determining a first node of a graph neural network (GNN) representing the first feature associated with the object and the second feature associated with the environment; determining a second node of the GNN representing the vehicle, wherein an edge connecting the first node and the second node represents a state of at least one of the vehicle or the object as a relative state; determining a first output based at least in part on the GNN representing a first predicted position of the object at a second time after the first time; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object.

[0223] O. The one or more non-transitory computer-readable media of claim N, wherein the vehicle is an autonomous vehicle that traverses an environment, and the operation further includes controlling the autonomous vehicle based at least in part on the trajectory of the object.

[0224] P. The one or more non-transitory computer-readable media described in clause N or O, wherein the operations further include determining a second output representing a second predicted position of the vehicle at a second time based at least in part on the GNN; receiving a reference trajectory of the vehicle through the environment from a planning component associated with the vehicle; updating a second node of the GNN based at least in part on the reference trajectory, wherein an edge connecting the first node and the second node represents a candidate position of the vehicle; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object and the candidate position of the vehicle.

[0225] Q. One or more non-transitory computer-readable media of any one of clauses N to P, wherein an edge connecting the first node and the second node represents a state of the object relative to the vehicle at a first time, and the operations further include determining a feature map associated with the environment based at least in part on the map data, determining a third feature associated with the environment based at least in part on a first predicted position of the object in the environment and the feature map, updating the first node of the GNN based at least in part on the third feature, and determining a second output representing a second predicted position of the object at a third time after the second time based at least in part on the GNN, and determining a first predicted trajectory of the object is based at least in part on the first predicted position of the object and the second predicted position of the object.

[0226] R. The non-transitory computer readable medium of one or more of N to Q terms, wherein the operations further include determining a first characteristic associated with the object based at least in part on data representing at least one of: an object pose, an object position, an object acceleration, an object velocity, an object size, an object type, and an object lighting condition.

[0227] S. The one or more non-transitory computer-readable media of any one of clauses N to R, wherein the operations further include determining a second feature associated with the environment based at least in part on data representing at least one of: static objects in the environment, dynamic objects in the environment, traffic signs in the environment, traffic lights in the environment, types of static or dynamic objects in the environment, lighting conditions for static or dynamic objects in the environment, and lane markers in the environment.

[0228] T. The one or more non-transitory computer-readable media of paragraph S, wherein the operations further include determining a second feature associated with the environment based at least in part on inputting the data into a convolutional neural network (CNN).

[0229] U. A system including one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving sensor data from sensors associated with a vehicle in an environment, the environment including an object; determining a first feature associated with the object at a first time based at least in part on the sensor data; determining a feature map representative of an area associated with the environment based at least in part on map data associated with the environment; determining a second feature associated with the environment at the first time based at least in part on a current position of the object in the environment and the feature map; determining nodes of a graph neural network (GNN) representing the first and second features; determining a first output representative of a first predicted position of the object outside the area of ​​the environment based at least in part on the GNN; determining a second predicted position of the object based at least in part on the first predicted position that is outside the area and the map data; and controlling the vehicle based at least in part on the second predicted position.

[0230] V. The system described in paragraph U, wherein the first output further includes first distribution data for the object, the first distribution data including a first predicted position of the object, and the operation further includes determining the first predicted position based at least in part on sampling from the first distribution data.

[0231] W. The operations further include determining a reference line based at least in part on a current position of the object in the environment and the map data, and determining a second predicted position of the object is based at least in part on projecting the first predicted position onto the reference line, in a U or V system.

[0232] X. The system of any one of paragraphs U to W, wherein the operations further include determining that a second predicted position of the object in the environment is outside a region associated with the feature map; determining a third feature associated with the environment based at least in part on the second predicted position of the object being outside the region, where the third feature is a null feature; updating a node of the GNN based at least in part on the third feature; determining a second output representing the third predicted position of the object outside the region of the environment based at least in part on the GNN; determining a fourth predicted position of the object based at least in part on the third predicted position and the map data; and controlling the vehicle based at least in part on the second predicted position of the object and the fourth predicted position of the object.

[0233] Y. The system of any one of paragraphs U to X, wherein a first distance between the current position of the object in the environment and the first predicted position of the object is the same as a second distance between the current position of the object in the environment and the second predicted position of the object.

[0234] Z. A method comprising: receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object; determining a first feature associated with the object based at least in part on the sensor data; determining a feature map representing a region of the environment based at least in part on map data associated with the environment; determining a second feature associated with the environment based at least in part on a current position of the object and the feature map; associating the first feature and the second feature with nodes of a graph structure; determining a first output representing a first predicted position of the object outside the region of the environment based at least in part on the graph structure; and determining a second predicted position of the object based at least in part on the first predicted position of the object and the map data.

[0235] AA. The method of paragraph Z, wherein the first output further includes first distribution data for the object, the first distribution data including a first predicted position of the object, and the method further includes determining the first predicted position of the object based at least in part on sampling from the first distribution data.

[0236] AB. The method of claim Z or AA, further comprising determining a reference line based at least in part on a current position of the object in the environment and map data, and determining a second predicted position of the object is based at least in part on projecting the first predicted position onto the reference line.

[0237] AC. The method of any one of paragraphs Z-AB, further comprising: determining that a second predicted position of the object in the environment is outside a region associated with the feature map; determining a third feature associated with the environment based at least in part on the second predicted position of the object being outside the region, where the third feature is a null feature; updating a node of the graph structure based at least in part on the third feature; determining a second output representing the third predicted position of the object outside the region of the environment based at least in part on the graph structure; determining a fourth predicted position of the object based at least in part on the third predicted position and the map data; and controlling the vehicle based at least in part on the second predicted position of the object and the fourth predicted position of the object.

[0238] AD. The method of any one of paragraphs Z to AC, wherein a first distance between a current position of the object in the environment and a first predicted position of the object is the same as a second distance between a current position of the object in the environment and a second predicted position of the object.

[0239] AE. The method of any one of paragraphs Z to AD, wherein determining a second predicted position of the object is based, at least in part, on a vehicle dynamics model.

[0240] The method of any one of paragraphs Z to AE, wherein the AF.region is a bounded region, and a portion of the feature map corresponding to the bounded region is used to plan a trajectory of a vehicle through the environment.

[0241] AG. The method of any one of paragraphs Z to AF, wherein the first feature is determined based at least in part on inputting a portion of the sensor data associated with the object into a first neural network, and the feature map is determined based at least in part on inputting a field of view of the environment into a second neural network different from the first neural network, the field of view of the environment being based at least in part on the sensor data, and the second feature is associated with a portion of a region of the environment corresponding to the object.

[0242] AH. One or more non-transitory computer readable media storing instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations including receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object, determining a first feature associated with the object based at least in part on the sensor data, determining a feature map representing a region of the environment based at least in part on map data associated with the environment, determining a second feature associated with the environment based at least in part on a current position of the object and the feature map, associating the first feature and the second feature with nodes of a graph structure, determining a first output representing a first predicted position of the object outside the region of the environment based at least in part on the graph structure, and determining a second predicted position of the object based at least in part on the first predicted position of the object and the map data.

[0243] AI. The one or more non-transitory computer-readable media described in paragraph AH, wherein the first output further includes first distribution data for the object, the first distribution data including a first predicted position of the object, and the operation further includes determining the first predicted position of the object based at least in part on sampling from the first distribution data.

[0244] AJ. One or more non-transitory computer-readable media as described in paragraphs AH or AI, wherein the operations further include determining a reference line based at least in part on a current position of the object in the environment and the map data, and determining a second predicted position of the object is based at least in part on projecting the first predicted position onto the reference line.

[0245] AK. One or more non-transitory computer-readable media of any one of paragraphs AH-AJ, wherein the operations include determining that a second predicted position of the object in the environment is outside a region associated with the feature map; determining a third feature associated with the environment based at least in part on the second predicted position of the object being outside the region, where the third feature is a null feature; updating a node of the graph structure based at least in part on the third feature; determining a second output representing the third predicted position of the object outside the region of the environment based at least in part on the graph structure; determining a fourth predicted position of the object based at least in part on the third predicted position and the map data; and controlling a vehicle based at least in part on the second predicted position of the object and the fourth predicted position of the object.

[0246] AL. One or more non-transitory computer-readable media of any one of paragraphs AH to AK, wherein a first distance between a current position of an object in the environment and a first predicted position of the object is the same as a second distance between a current position of the object in the environment and a second predicted position of the object.

[0247] AM. One or more non-transitory computer-readable media of any one of paragraphs AH-AL, wherein determining a second predicted position of the object is based, at least in part, on a vehicle dynamics model.

[0248] AN. One or more non-transitory computer-readable media described in any one of paragraphs AH to AM, wherein the region is a bounded region and a portion of the feature map corresponding to the bounded region is used to plan a trajectory of a vehicle through the environment.

[0249] AO. A system having one or more processors, the system, when executed by the one or more processors, includes the steps of: receiving sensor data from sensors associated with a vehicle in an environment, the environment including an object; determining a first feature associated with one of the object or the vehicle based at least in part on the sensor data; determining a second feature associated with the environment based at least in part on map data associated with the environment; determining graph nodes of a graph neural network (GNN) based on the first feature and the second feature; and determining an inference associated with the node based at least in part on the GNN. 1. A system comprising: one or more non-transitory computer-readable media storing computer-executable instructions to perform operations including: determining a first output representing distribution data, the distribution data including a predicted position of an object or vehicle in an environment; determining a sampling technique for sampling the distribution data based at least in part on at least one of a classification type of a graph node, a predicted position of the distribution data, a distribution type of the distribution data, or an input associated with the distribution data; determining a first predicted position of the object or vehicle based at least in part on the sampling technique; and controlling a vehicle based at least in part on the first predicted position.

[0250] AP. The system described in paragraph AO, wherein the operation further includes one of: determining that the classification type of the graph node indicates that the graph node is associated with an object, and the sampling technique includes a first sampling technique; or determining that the classification type of the graph node indicates that the graph node is associated with a vehicle, and the sampling technique includes a second sampling technique that is different from the first sampling technique.

[0251] AQ. The system described in paragraph AO or AP, wherein the operations further include determining a boundary area of ​​the environment used to plan the vehicle's trajectory, determining, based at least in part on the distribution data, that a predicted location of the distribution data is outside the boundary area, determining a sampling technique, based at least in part on the predicted location of the distribution data being outside the boundary area, and determining a first predicted location of the object, based at least in part on the sampling technique and map data associated with the environment.

[0252] The system of any one of paragraphs AO to AQ, wherein the AR. operation further includes one of a first operation of determining that a distribution type of the distribution data is a first distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using a sampling technique, the sampling technique including a Gaussian sampling technique, or a second operation of determining that a distribution type of the distribution data is a second distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using the sampling technique, the first predicted position being an average of the predicted positions of the distribution data.

[0253] The system of any one of paragraphs AO to AR, wherein the AS. vehicle is an autonomous vehicle and the operation further includes receiving input from a planning component of the autonomous vehicle, and the first predicted position of the object or vehicle is based at least in part on the input data.

[0254] AT. A method comprising: receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object; associating a first feature associated with one of the object or the vehicle and a second feature associated with the environment with nodes of a graph structure; determining a first output representing distribution data associated with the node based at least in part on the graph structure, the distribution data including a predicted position of the object or vehicle in the environment; determining a sampling technique for sampling the distribution data based at least in part on at least one of a classification type of the node, a predicted position of the distribution data, a distribution type of the distribution data, or an input associated with the distribution data; determining a first predicted position of the object or vehicle based at least in part on the sampling technique; and determining a predicted trajectory of the object or vehicle based at least in part on the first predicted position.

[0255] The method of paragraph AU.AT, further comprising one of: determining that the classification type of the node indicates that the node is associated with an object, and the sampling technique includes a first sampling technique; or determining that the classification type of the node indicates that the node is associated with a vehicle, and the sampling technique includes a second sampling technique that is different from the first sampling technique.

[0256] The method of claim AT or AU, further comprising: determining a boundary region of an environment used to plan a trajectory of the AV vehicle; determining, based at least in part on the distribution data, that a predicted position of the distribution data is outside the boundary region; determining a sampling technique, based at least in part on the predicted position of the distribution data being outside the boundary region; and determining a first predicted position of the object, based at least in part on the sampling technique and map data associated with the environment.

[0257] AW. The method of any one of paragraphs AT to AV, further comprising one of: a first operation including determining that a distribution type of the distribution data is a first distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using a sampling technique, the sampling technique including a Gaussian sampling technique; or a second operation including determining that the distribution type of the distribution data is a second distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using the sampling technique, the first predicted position being an average of the predicted positions of the distribution data.

[0258] AX. The method of any one of paragraphs AT to AW, wherein the vehicle is an autonomous vehicle and the method further includes receiving input from a planning component of the autonomous vehicle, and wherein the first predicted position of the object or vehicle is based at least in part on the input data.

[0259] AY. The method of any one of paragraphs AT to AX, further comprising receiving an input, the input data indicating a candidate position of an object or vehicle in the environment that is different from the predicted position, and determining a predicted trajectory of the object or vehicle based on the selected position.

[0260] AZ. The method of any one of clauses AT-AY, where the sampling technique is a first sampling technique and a first predicted location is associated with a first time, the method further including: updating a node having a third feature associated with the environment based at least in part on the first predicted location of the object or vehicle; determining a second output representing distribution data associated with the node based at least in part on the graph structure, the distribution data including a predicted location of the object or vehicle in the environment at a second time after the first time; determining a second sampling technique for sampling the distribution data, the second sampling technique being different from the first sampling technique; determining a second predicted location in the environment of the object or vehicle at the second time based at least in part on the second sampling technique; and determining a predicted trajectory based at least in part on the first predicted location and the second predicted location.

[0261] BA. In a method according to the present invention, the object is a first object, the node is a first node associated with the first object, the distribution data is first distribution data associated with the first node at a first time, and the sampling technique is a first sampling technique, the method comprising: associating a third feature associated with a second object in the vehicle or environment and a second feature associated with the environment to a second node in a graph structure; and determining a second output representing second distribution data associated with the second node based at least in part on the graph structure, the second distribution data including a second predicted position of the vehicle or the second object in the environment at the first time; The method of any one of paragraphs AT-AZ, further comprising: determining a first sampling technique for sampling the first distribution data based at least in part on the class type; determining a second sampling technique for sampling the second distribution data based at least in part on the first classification type of the second node, the second sampling technique being different from the first sampling technique; determining a second predicted position of the vehicle or second object at the first time based at least in part on the second sampling technique; and determining a predicted trajectory of the first object or vehicle based at least in part on the first predicted position or the second predicted position.

[0262] BB. One or more non-transitory computer readable media storing instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations including receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object; associating a first feature associated with one of the object or the vehicle and a second feature associated with the environment with a node of a graph structure; determining a first output representing distribution data associated with the node based at least in part on the graph structure, the distribution data including a predicted position of the object or the vehicle in the environment; determining a sampling technique for sampling the distribution data based at least in part on at least one of a classification type of the node, a predicted position of the distribution data, a distribution type of the distribution data, or an input associated with the distribution data; determining a first predicted position of the object or vehicle based at least in part on the sampling technique; and determining a predicted trajectory of the object or vehicle based at least in part on the first predicted position.

[0263] BC. The one or more non-transitory computer-readable media of paragraph BB, wherein the operation further includes one of: determining that the classification type of the node indicates that the node is associated with an object, and the sampling technique includes a first sampling technique; or determining that the classification type of the node indicates that the node is associated with a vehicle, and the sampling technique includes a second sampling technique that is different from the first sampling technique.

[0264] BD. The one or more non-transitory computer-readable media of paragraphs BB or BC, wherein the operations further include determining a boundary region of an environment used to plan a trajectory of the vehicle, determining, based at least in part on the distribution data, that a predicted position of the distribution data is outside the boundary region, determining a sampling technique, based at least in part on the predicted position of the distribution data being outside the boundary region, and determining a first predicted position of the object, based at least in part on the sampling technique and map data associated with the environment.

[0265] BE. The one or more non-transitory computer-readable media of any one of clauses BB to BD, wherein the operations further include one of: a first operation including determining that a distribution type of the distribution data is a first distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using a sampling technique, where the sampling technique includes a Gaussian sampling technique; or a second operation including determining that a distribution type of the distribution data is a second distribution type and determining a first predicted position of the object or vehicle based at least in part on sampling the distribution data using the sampling technique, where the first predicted position is an average of the predicted positions of the distribution data.

[0266] BF. One or more non-transitory computer-readable media of any one of paragraphs BB to BE, wherein the vehicle is an autonomous vehicle and the operation further includes receiving input from a planning component of the autonomous vehicle, and the first predicted position of the object or vehicle is based at least in part on the input data.

[0267] BG. One or more non-transitory computer-readable media described in any one of paragraphs BB to BF, wherein the operation further includes receiving input, the input data indicating a candidate position of an object or vehicle in the environment that is different from the predicted position, and determining a predicted trajectory of the object or vehicle based on the selected position.

[0268] BH. The one or more non-transitory computer-readable media of any one of paragraphs BB-BG, wherein the sampling technique is a first sampling technique, and the first predicted position is associated with a first time, and the operations further include updating a node having a third feature associated with the environment based at least in part on the first predicted position of the object or vehicle; determining a second output representing distribution data associated with the node based at least in part on the graph structure, the distribution data including a predicted position of the object or vehicle in the environment at a second time after the first time; determining a second sampling technique for sampling the distribution data, the second sampling technique being different from the first sampling technique; determining a second predicted position in the environment of the object or vehicle at the second time based at least in part on the second sampling technique; and determining a predicted trajectory based at least in part on the first predicted position and the second predicted position.

[0269] 1. A BI. system comprising: one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving sensor data from sensors associated with a vehicle in an environment at a first time, the environment including an object; determining a first feature associated with the object based at least in part on the sensor data; determining a second feature associated with the environment based at least in part on map data associated with the environment; associating the first feature associated with the object and the second feature associated with the environment with a node of a graph neural network (GNN); determining a first output representing first distribution data including measured positions of objects satisfying the criterion based at least in part on the GNN and a criterion; determining a first predicted position of the object at a second time after the first time based at least in part on sampling from the first distribution data; and controlling the vehicle based on the first predicted position of the object.

[0270] BJ. The system described in paragraph BI, wherein the sensor is associated with an autonomous vehicle traversing the environment, and the operations further include receiving a trajectory of the vehicle through the environment from a planning component associated with the autonomous vehicle, and determining a criterion based at least in part on the trajectory of the autonomous vehicle.

[0271] BK. The system of paragraphs BI or BJ, wherein the operations further include determining an intent associated with the object based at least in part on a first characteristic associated with the object and a second characteristic associated with the environment, and determining a criterion based at least in part on the intent associated with the object.

[0272] BL. The system of any one of paragraphs BI to BK, wherein operations further include receiving map data associated with the environment; determining rules associated with the environment based at least in part on the map data and the sensor data, the rules including at least one of traffic signs, traffic lights, zones in the environment, or speed limits; and determining criteria based at least in part on the rules associated with the environment.

[0273] BM. The system of any one of paragraphs BI to BL, wherein the criteria are based at least in part on object characteristics associated with the object, the object characteristics including at least one of a turning radius associated with the object, a top speed associated with the object, an acceleration limit associated with the object, a lane change maneuver associated with the object, a turning maneuver associated with the object, or an adverse behavior maneuver associated with the object.

[0274] BN. A method comprising: receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object at a first time; associating a first feature associated with the object and a second feature associated with the environment with nodes of a graph structure; determining a first output representing first distribution data including predicted positions of objects satisfying the criterion based at least in part on the graph structure and a criterion; determining a first predicted position of the object at a second time after the first time based at least in part on sampling from the first distribution data; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object.

[0275] BO. The method of claim BN, wherein the vehicle is an autonomous vehicle traversing the environment, and the method further includes receiving a trajectory of the vehicle through the environment from a planning component associated with the autonomous vehicle, and determining a criterion based at least in part on the trajectory of the autonomous vehicle.

[0276] BP. The method of claim BN or BO, further comprising: determining an intent associated with the object based at least in part on a first characteristic associated with the object and a second characteristic associated with the environment; and determining a criterion based at least in part on the intent associated with the object.

[0277] BQ. The method of any one of paragraphs BN-BP, further including receiving map data associated with the environment; determining rules associated with the environment based at least in part on the map data and the sensor data, the rules including at least one of a stop sign, a traffic light, a zone in the environment, or a speed limit; and determining criteria based at least in part on the rules associated with the environment.

[0278] BR. The method of any one of paragraphs BN to BQ, further comprising determining the criterion based at least in part on object characteristics associated with the object, the object characteristics including at least one of a turning radius associated with the object, a top speed associated with the object, an acceleration limit associated with the object, a lane change maneuver associated with the object, a turning maneuver associated with the object, or an adverse behavior maneuver associated with the object.

[0279] BS. The method of any one of clauses BN to BR, further comprising determining a criterion based at least in part on a first characteristic of the object, the first characteristic comprising at least one of a current pose of the object, a current position of the object, a current acceleration of the object, a current velocity of the object, a size of the object, a type of the object, or a lighting condition of the object.

[0280] BT. The method of any one of paragraphs BN-BS, where the object is a first object and the criterion is a first criterion, the method further including determining a third feature associated with a second object in the environment based at least in part on the sensor data; associating the third feature associated with the second object and the second feature associated with the environment to a second node of a graph structure; determining a second criterion based at least in part on the first predicted position of the first object; determining a second output representing second distribution data for the second object based at least in part on the graph structure and the second criterion, the second distribution data including a second predicted position of the second object that satisfies the second criterion; determining a second predicted position of the second object at a second time based at least in part on sampling from the second distribution data; and determining a second predicted trajectory of the second object based at least in part on the second predicted position of the second object and the first predicted trajectory of the first object.

[0281] BU. The method of claim BT, wherein the vehicle is an autonomous vehicle traversing the environment, the method further including determining a third predicted trajectory of the autonomous vehicle based at least in part on the first predicted trajectory of the first object and the second predicted trajectory of the second object, and controlling the autonomous vehicle based at least in part on the third predicted trajectory.

[0282] BV. One or more non-transitory computer readable media storing instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations including receiving sensor data from a sensor associated with a vehicle in an environment, the environment including an object at a first time; associating a first feature associated with the object and a second feature associated with the environment with nodes of a graph structure; determining a first output representing first distribution data including a predicted position of the object satisfying the criterion based at least in part on the graph structure and a criterion; determining a first predicted position of the object at a second time after the first time based at least in part on sampling from the first distribution data; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object.

[0283] BW. The one or more non-transitory computer-readable media described in section BV, wherein the vehicle is an autonomous vehicle traversing the environment, and the operations further include receiving a trajectory of the vehicle through the environment from a planning component associated with the autonomous vehicle, and determining a criterion based at least in part on the trajectory of the autonomous vehicle.

[0284] BX. One or more non-transitory computer-readable media as described in paragraphs BV or BW, wherein the operations further include determining an intent associated with the object based at least in part on a first characteristic associated with the object and a second characteristic associated with the environment, and determining a criterion based at least in part on the intent associated with the object.

[0285] BY. One or more non-transitory computer-readable media of any one of paragraphs BV to BX, wherein the operations further include receiving map data associated with the environment; determining rules associated with the environment based at least in part on the map data and the sensor data, the rules including at least one of a stop sign, a traffic light, a zone in the environment, or a speed limit; and determining criteria based at least in part on the rules associated with the environment.

[0286] BZ. The one or more non-transitory computer-readable media of any one of paragraphs BV to BY, wherein the operation further includes determining the criterion based at least in part on object characteristics associated with the object, the object characteristics including at least one of a turning radius associated with the object, a top speed associated with the object, an acceleration limit associated with the object, a lane change maneuver associated with the object, a turning maneuver associated with the object, or an adverse behavior maneuver related to the object.

[0287] CA. The one or more non-transitory computer-readable media of any one of paragraphs BV to BZ, wherein the operations further include determining a criterion based at least in part on a first characteristic of the object, the first characteristic including at least one of a current pose of the object, a current position of the object, a current acceleration of the object, a current velocity of the object, a size of the object, a type of the object, or a lighting condition of the object.

[0288] CB. The one or more non-transitory computer-readable media of any one of paragraphs BV-CA, wherein the object is a first object and the criterion is a first criterion, and the operations further include determining a third feature associated with a second object in the environment based at least in part on the sensor data; determining a second criterion based at least in part on the first predicted position of the first object after associating the third feature associated with the second object and the second feature associated with the environment to a second node of the graph structure; determining a second output representing second distribution data for the second object based at least in part on the graph structure and the second criterion, the second distribution data including a second predicted position of the second object that satisfies the second criterion; determining a second predicted position of the second object at a second time based at least in part on sampling from the second distribution data; and determining a second predicted trajectory of the second object based at least in part on the second predicted position of the second object and the first predicted trajectory of the first object.

[0289] Although the example sections above are described with respect to one particular implementation, it should be understood in the context of this specification that the subject matter of the example sections may be implemented via a method, apparatus, system, computer readable medium, and / or other implementations. Furthermore, any of the example sections A-CB may be implemented alone or in combination with any one or more of the other example sections A-CB.

[0290] (Conclusion) One or more examples of the techniques described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the techniques described herein.

[0291] In describing the examples, reference is made to the accompanying drawings, which form a part hereof, and which illustratively show specific examples of the claimed subject matter. It is to be understood that other examples may be used and that modifications or alterations, such as structural changes, may be made. Such examples, modifications or alterations do not necessarily depart from the intended scope of the claimed subject matter. Although the procedures described herein may be shown in a certain order, in some cases the order may be changed so that certain inputs are provided at different times or in a different order without changing the functionality of the systems and methods described. Also, the procedures disclosed may be performed in a different order. Furthermore, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to changing the order, it is also possible to decompose the calculations into sub-calculations that have the same results.

Claims

1. 1. A method comprising: receiving sensor data from sensors associated with a vehicle in an environment, the environment including objects; determining a first feature associated with the object at a first time based at least in part on the sensor data; and determining a second characteristic associated with the environment based at least in part on the sensor data or map data associated with the environment; determining a first node of a graph neural network (GNN) representing the first feature associated with the object and the second feature associated with the environment; determining a second node of the GNN representing the vehicle, wherein an edge connecting the first node and the second node represents a state of at least one of the vehicle or the object as a relative state; determining a first output representing a first predicted position of the object at a second time after the first time based at least in part on the GNN; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object; A method for providing the above.

2. 10. The method of claim 1, wherein the vehicle is an autonomous vehicle traversing the environment, the method further comprising controlling the autonomous vehicle based at least in part on the trajectory of the object.

3. determining a second output representative of a second predicted position of the vehicle at the second time based at least in part on the GNN; and receiving a reference trajectory of the vehicle through the environment from a planning component associated with the vehicle; updating the second node of the GNN based at least in part on the reference trajectory, wherein the edge connecting the first node and the second node represents a candidate position for the vehicle; determining the first predicted trajectory of the object based at least in part on the first predicted position of the object and the candidate positions of the vehicle; The method of claim 1 further comprising:

4. The edge connecting the first node and the second node represents the state of the object relative to the vehicle at the first time, and the method further comprises: determining a feature map associated with the environment based at least in part on the map data; and determining a third feature associated with the environment based at least in part on the first predicted position of the object in the environment and the feature map; and updating the first node of the GNN based at least in part on the third characteristic; and determining a second output representing a second predicted position of the object at a third time after the second time based at least in part on the GNN; and Furthermore, determining the first predicted trajectory of the object is based at least in part on the first predicted position of the object and the second predicted position of the object; The method of claim 1.

5. the pose of the object; the position of the object; the acceleration of the object, the velocity of the object, the size of the object; the type of the object, or the lighting conditions of the object; The method of claim 1 , further comprising determining the first characteristic associated with the object based at least in part on data representing at least one of:

6. static objects in the environment; dynamic objects in the environment; traffic signs within said environment; a traffic light in said environment; the type of the static or dynamic objects in the environment; the lighting conditions of the static or dynamic objects in the environment, or lane markings within said environment; The method of claim 1 , further comprising determining the second characteristic associated with the environment based at least in part on data representing at least one of:

7. the first feature is determined based at least in part on inputting data associated with the object into a first neural network, the data being based at least in part on the sensor data; the second characteristics are determined based at least in part on inputting a view of the environment into a second neural network different from the first neural network, the view of the environment being based at least in part on the sensor data; the second feature is associated with a portion of the environment corresponding to the object; The method of claim 1.

8. 8. One or more non-transitory computer-readable media storing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to perform the method of any one of claims 1 to 7.

9. 1. A system comprising: one or more processors; One or more non-transitory computer-readable media that, when executed by the one or more processors, cause the one or more processors to: receiving sensor data from sensors associated with a vehicle in an environment, the environment including objects; determining a first feature associated with the object at a first time based at least in part on the sensor data; and determining a second characteristic associated with the environment based at least in part on the sensor data or map data associated with the environment; determining a first node of a graph neural network (GNN) representing the first feature associated with the object and the second feature associated with the environment; determining a second node of the GNN representing the vehicle, wherein an edge connecting the first node and the second node represents a state of at least one of the vehicle or the object as a relative state; determining a first output representing a first predicted position of the object at a second time after the first time based at least in part on the GNN; and determining a first predicted trajectory of the object based at least in part on the first predicted position of the object; the one or more non-transitory computer-readable media storing computer-executable instructions for performing operations comprising: A system comprising:

10. 10. The system of claim 9, wherein the vehicle is an autonomous vehicle that traverses the environment, and the action further comprises controlling the autonomous vehicle based at least in part on the trajectory of the object.

11. The operation is determining a second output representative of a second predicted position of the vehicle at the second time based at least in part on the GNN; and receiving a reference trajectory of the vehicle through the environment from a planning component associated with the vehicle; updating the second node of the GNN based at least in part on the reference trajectory, wherein the edge connecting the first node and the second node represents a candidate position for the vehicle; determining the first predicted trajectory of the object based at least in part on the first predicted position of the object and the candidate positions of the vehicle; The system of claim 9 or 10, further comprising:

12. The edge connecting the first node and the second node represents the state of the object relative to the vehicle at the first time, and the action is determining a feature map associated with the environment based at least in part on the map data; and determining a third feature associated with the environment based at least in part on the first predicted position of the object in the environment and the feature map; and updating the first node of the GNN based at least in part on the third characteristic; and determining a second output representing a second predicted position of the object at a third time after the second time based at least in part on the GNN; and Furthermore, determining the first predicted trajectory of the object is based at least in part on the first predicted position of the object and the second predicted position of the object; The system of claim 9.

13. The operation is the pose of the object; the position of the object; the acceleration of the object, the velocity of the object, the size of the object; the type of the object, or the lighting conditions of the object; 10. The system of claim 9, further comprising determining the first characteristic associated with the object based at least in part on data representing at least one of:

14. The operation is static objects in the environment; dynamic objects in the environment; traffic signs within said environment; a traffic light in said environment; the type of the static or dynamic objects in the environment; the lighting conditions of the static or dynamic objects in the environment, or lane markings within said environment; The system of claim 9 , further comprising determining the second characteristic associated with the environment based at least in part on data representing at least one of:

15. 15. The system of claim 14, wherein the operations further comprise determining the second feature associated with the environment based at least in part on inputting the data into one of a convolutional neural network (CNN) or a recurrent neural network (RNN).

Images