Target-Based Motion Prediction

Abstract

An exemplary aspect of the present disclosure relates to an exemplary computer-implemented method for predicting the intentions of actors in an environment. The exemplary method includes obtaining state data related to a plurality of actors in the environment and map data representing a plurality of lanes in the environment. The method includes determining a plurality of potential goals for each actor based on the state data and the map data. The method includes processing the state data, the map data, and the plurality of potential goals with a machine learning predictive model to determine (i) predicted goals for each actor of the plurality of actors, (ii) predicted interactions between each actor of the plurality of actors and other actors based on the predicted goals, and (iii) continuous trajectories for each actor based on the predicted goals.

Description

[Technical Field]

[0001] Priority claims This application claims the benefit of U.S. Patent Application No. 18 / 147,316, filed December 28, 2022, and U.S. Patent Application No. 18 / 471,960, filed September 21, 2023, which are incorporated herein by reference in their entireties. [Background technology]

[0002] An autonomous platform can process data to perceive the environment it must navigate in. For example, an autonomous vehicle can use various sensors to perceive its surroundings and identify objects around the autonomous vehicle. The autonomous vehicle can then identify an appropriate path through the perceived environment and navigate along that path with no or minimal human intervention. Summary of the Invention

[0003] This disclosure describes improved intent prediction techniques that can be leveraged by autonomous platforms for motion prediction and ultimately motion planning. The improved intent prediction techniques include machine learning models (e.g., graph neural networks) trained to generate multiple discrete intent and continuous motion outputs based on past actor observations and map geometry of a traffic scene. The outputs include (i) goal probabilities (e.g., the probability that an actor on a road will follow a nominal path), (ii) interaction probabilities (e.g., the probability that an actor will yield or reverse yield to another actor), and (iii) a continuous goal-based trajectory of the actor. For example, an autonomous platform, such as an autonomous vehicle, can use these outputs to more accurately predict the future movements of actors in the environment while planning its movements. In some cases, additional machine learning models (e.g., typified graph neural networks) can be used to adjust the outputs according to the autonomous platform's expected movements. The techniques described herein can improve the speed, efficiency, and accuracy of predicting future movements of dynamic actors in traffic scenes, thereby improving the decision and response times of dynamic actors when planning the movements of autonomous platforms.

[0004] More specifically, the present disclosure relates to a machine learning predictive model for predicting future movements of actors in a traffic scene based on state data and map data about the environment. Actors include both autonomous platforms and other dynamic objects in the traffic scene. The state data may include current and past observations, such as position, speed, and direction of movement, for each actor in the scene. The map data may identify multiple lanes in the traffic scene and different lane characteristics for each of the multiple lanes. The model may determine multiple potential goals for each actor in the traffic scene based on the actor's current state and the map data. Each goal may include a short-term goal (e.g., a potential goal within the next 5 seconds) and a target path to the target (e.g., multiple waypoints along one or more traffic lanes) available to each actor based on the actor's location in the traffic scene. The machine learning predictive model processes the state data and map data to determine (i) the probability that the actor will follow the target path and / or (ii) the probability that the actor will interact with other actors in the traffic scene while following the target path. Furthermore, the model can determine (iii) the continuous trajectory of the actor, which is conditioned according to the goal path.

[0005] The machine learning predictive model may include a graph neural network having a plurality of nodes and edges. The plurality of nodes may include one or more goal nodes of a goal node type and one or more actor nodes of an actor node type. The plurality of edges may include one or more actor-goal edges of an actor-goal edge type and one or more goal-goal edges of a goal-goal edge type. An actor-goal edge may connect a particular actor to a potential goal of that actor, and a goal-goal edge may connect two goals associated with a "shared space" (e.g., a space where two corresponding goal paths intersect), which may lead to an interaction between two actors that respectively achieve two goals.

[0006] Multiple nodes and edges may include feature representations encoded using features different from the state data and map data based on their respective node and edge types. For example, an actor node may include an actor representation encoded based on a specific actor's current state and one or more past states. A goal node may include a goal representation encoded based on waypoint information for a corresponding goal path. An actor-goal edge may include an actor-goal representation encoded based on actor state information in a path tangent frame. Furthermore, a goal-goal edge may include a goal-goal representation encoded based on actor state information for two actors that may achieve two goals related to a shared space. Message forwarding may be performed in several rounds to update the feature representations based on information from neighboring nodes and edges. Outputs may be determined from the updated feature representations.

[0007] The machine learning predictive model can be trained using labeled training data. The labeled training data can include auto-labeled ground-truth information that identifies (i) a ground-truth goal path followed by one or more actors in a traffic scene, (ii) whether an interaction occurred, and (iii) the type of interaction that occurred (e.g., which actor yielded). The ground-truth information can be determined algorithmically based on future sensor observations of the traffic scene. The machine learning predictive model can be trained using a loss function that minimizes the distance between the output and the ground-truth information.

[0008] Aspects of the present disclosure may provide various technical improvements to robotics, machine learning, and intent prediction technologies. The improved intent prediction technologies described herein may provide a technical solution to the technical problem of robots predicting the movement of dynamic objects in real time. As a result, the systems and methods of the present disclosure may provide an improved approach to planning autonomous driving movements.

[0009] The systems and methods described herein may provide substantial improvements to robotics, machine learning, and robot intent prediction technologies by accumulating and leveraging newly available information, such as probabilistic goal-based paths for dynamic objects, probabilistic interaction relationships between dynamic objects, and continuous trajectories conditioned according to the probabilistic goal-based paths and / or interaction relationships. Unlike traditional intent prediction technologies, the systems and methods described herein output several discrete intents and continuous motion outputs that can be used to train machine learning models to accurately predict the motion of dynamic objects in an environment. In this way, the systems and methods described herein can improve the speed, efficiency, and accuracy of future object position predictions. This ultimately enhances the capabilities of robotics and machine learning technologies by enabling autonomous platforms to predict the future motion of objects in an environment. Ultimately, the techniques disclosed herein improve motion planning and road driving performance for various configurations of robots or autonomous vehicle technologies by implementing more accurate, robust, and efficient object intent prediction techniques.

[0010] For example, in one aspect, the present disclosure provides an exemplary computer-implemented method. The exemplary method includes obtaining state data associated with a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The method includes determining, based on the state data and the map data, a plurality of potential targets, including at least one potential target, for each of the plurality of actors. The potential targets include a location of a potential destination within the environment and a target route to the potential destination location. The method includes processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) a predicted target for each of the plurality of actors, (ii) a predicted interaction between the each of the plurality of actors and other actors based on the predicted target, and (iii) a continuous trajectory for the each of the actors based on the predicted target. The method includes initiating movement of the autonomous vehicle based on the predicted target, the predicted interaction, or the continuous trajectory for the each of the actors.

[0011] In some embodiments of the exemplary method, the method includes determining a probability of each potential target for the individual actor, and the predicted targets include the individual potential targets having the highest probabilities.

[0012] In some embodiments of the example method, the state data is associated with at least one of a localization system or an inertial measurement unit of the autonomous vehicle.

[0013] In some embodiments of the exemplary method, the target path of the potential target includes a nominal path defined in the map data.

[0014] In some embodiments of the exemplary method, the map data includes a plurality of map features associated with the potential destination, and the plurality of map features includes a plurality of waypoints for the nominal route defined in the map data.

[0015] In some embodiments of the exemplary method, the predicted interactions are further based on potential goals for other actors in the environment that intersect with the predicted goals for the individual actor.

[0016] In some embodiments of the exemplary method, the predicted interactions include probability distributions for one or more interaction types between the individual actor and the other actors.

[0017] In some embodiments of the exemplary method, the predicted probability of interaction between the individual actor and the other actor is based on one or more distances between the individual actor and a shared space associated with the predicted goal and the potential goal of the other actor.

[0018] In some embodiments of the exemplary method, the state data includes a plurality of actor states associated with the individual actor, the plurality of actor states indicating one or more positions, one or more velocities, or one or more headings for the individual actor at a current time and one or more past times.

[0019] In some embodiments of the exemplary method, processing the state data, the map data, and the multiple potential targets with a machine learning predictive model to determine (i) a predicted goal for each of the multiple actors, (ii) a predicted interaction between the each of the multiple actors and other actors based on the predicted goal, and (iii) a continuous trajectory for the each of the actors based on the predicted goal includes: processing the state data, the map data, and the multiple potential targets with a first portion of the machine learning predictive model to generate a plurality of feature embeddings corresponding to the multiple actors and the multiple potential targets; and processing the plurality of feature embeddings to determine the predicted goal for the each of the actors, the predicted interaction between the each of the actors and the other actors based on the predicted goal, and the continuous trajectory for the each of the actors based on the predicted goal.

[0020] In some embodiments of the exemplary method, processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the each of the plurality of actors and other actors based on the predicted targets, and (iii) continuous trajectories for the each of the actors based on the predicted targets includes: performing one or more rounds of message passing to generate a plurality of updated feature embeddings; and determining the predicted targets, the predicted interactions, and the continuous trajectories for the each of the actors based on the plurality of updated feature embeddings.

[0021] In some embodiments of the exemplary method, the machine learning predictive model includes a graph neural network including a plurality of nodes and a plurality of edges, wherein the plurality of nodes includes: (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors; and (ii) a plurality of goal nodes corresponding individually to each potential goal of the plurality of potential goals; and the plurality of edges includes: (iii) one or more actor-to-goal edges connecting each actor node with each goal node; and (iv) one or more goal-to-goal edges connecting at least two goal nodes of the plurality of goal nodes.

[0022] In some embodiments of the exemplary method, the multiple feature embeddings include an actor feature embedding corresponding to an actor node, a target feature embedding corresponding to a target node, an actor-target feature embedding corresponding to an actor-target edge, and a target-target feature embedding corresponding to a target-target edge.

[0023] In some embodiments of the exemplary method, processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) a predicted target for each of the plurality of actors, (ii) a predicted interaction between the each of the plurality of actors and other actors based on the predicted target, and (iii) a continuous trajectory for the each of the actors based on the predicted target includes processing at least one of the actor feature embeddings or the actor-target feature embeddings to determine the predicted target for the each of the actors, and processing the target-target feature embeddings to determine the predicted interaction between the each of the actors and the other actors based on the predicted target.

[0024] In some embodiments of the example method, the method includes receiving vehicle motion data for the autonomous vehicle, the vehicle motion data indicating potential motion of the autonomous vehicle.

[0025] In some embodiments of the example method, the method includes generating a conditional feature embedding based on the potential movement of the autonomous vehicle.

[0026] In some embodiments of the example method, the method further includes concatenating a conditional feature embedding with at least one of the plurality of feature embeddings.

[0027] In some embodiments of the exemplary method, the method further includes determining the predicted goals, the predicted interactions, and the continuous trajectories for the individual actors by processing the plurality of feature embeddings with a machine learning conditional prediction model.

[0028] For example, in one aspect, the present disclosure provides an exemplary computing system. The computing system includes one or more processors and one or more non-transitory computer-readable media storing executable instructions to enable the one or more processors to perform operations. The operations include acquiring state data related to a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include determining a plurality of potential targets, including at least one potential target, for each of the plurality of actors based on the state data and the map data. The potential targets include locations of potential destinations within the environment and target routes to the locations of the potential destinations. The operations include processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between each of the plurality of actors and other actors based on the predicted targets, and (iii) continuous trajectories for the each of the actors based on the predicted targets. The action includes initiating movement of the autonomous vehicle based on the predicted goal, the predicted interaction, or the continuous trajectory for the individual actor.

[0029] In some embodiments of the exemplary computing system, processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) a predicted target for each of the plurality of actors, (ii) a predicted interaction between the each of the plurality of actors and other actors based on the predicted targets, and (iii) a continuous trajectory for the each of the actors based on the predicted targets further includes determining a probability of each potential target for the each of the actors, wherein the predicted target includes the each of the potential targets having the highest probability.

[0030] In some embodiments of the exemplary computing system, the target route of the potential target comprises a nominal route defined in the map data.

[0031] In some embodiments of the exemplary computing system, the map data includes a plurality of map features associated with the potential destination, the plurality of map features including a plurality of waypoints for the nominal route defined in the map data.

[0032] For example, in one aspect, the present disclosure provides an exemplary control system for an autonomous vehicle. The control system includes one or more processors and one or more non-transitory computer-readable media storing executable instructions to enable the one or more processors to perform operations. The operations include acquiring state data related to a plurality of actors in an environment of the autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include determining a plurality of potential targets, including at least one potential target, for each of the plurality of actors based on the state data and the map data. The potential targets include locations of potential destinations within the environment and target routes to the locations of the potential destinations. The operations include processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the each of the plurality of actors and other actors based on the predicted targets, and (iii) continuous trajectories for the each of the actors based on the predicted targets. The action includes initiating movement of the autonomous vehicle based on the predicted goal, the predicted interaction, or the continuous trajectory for the individual actor.

[0033] In some embodiments of the exemplary control system, processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) a predicted target for each of the plurality of actors, (ii) a predicted interaction between the each of the plurality of actors and other actors based on the predicted targets, and (iii) a continuous trajectory for the each of the actors based on the predicted targets further includes determining a probability of each potential target for the each of the actors, wherein the predicted target includes the each of the potential targets having the highest probability.

[0034] For example, in one aspect, the present disclosure provides an exemplary computer-implemented method. The computer-implemented method includes: (a) obtaining state data associated with a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in the environment; (b) determining, based on the state data and the map data, a plurality of potential goals including at least one potential goal for each of the plurality of actors, the potential goals including a potential destination location within the environment and a desired route to the potential destination location; (c) generating a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential goals by processing the state data, the map data, and the plurality of potential goals with a machine learning predictive model; (d) processing the plurality of feature embeddings to determine predicted goals for each of the plurality of actors and a continuous trajectory for the each of the actors based on the predicted goals; and (e) initiating movement of the autonomous vehicle based on the predicted goals and the continuous trajectory for the each of the actors.

[0035] In some embodiments of the exemplary computer-implemented method, the machine learning predictive model includes the graph neural network having the individual nodes corresponding to the multiple actors and the multiple potential targets.

[0036] In some embodiments of the exemplary computer-implemented method, the graph neural network further includes one or more actor-goal edges respectively connecting each actor node with each goal node.

[0037] In some embodiments of the exemplary computer-implemented method, the graph neural network further includes one or more goal-to-goal edges individually connecting at least two goal nodes of the individual nodes corresponding to the plurality of potential goals.

[0038] In some embodiments of the exemplary computer-implemented method, the plurality of feature embeddings include an actor feature embedding corresponding to an actor node, a goal feature embedding corresponding to a goal node, an actor-goal feature embedding corresponding to an actor-goal edge, and a goal-goal feature embedding corresponding to a goal-goal edge.

[0039] In some embodiments of the exemplary computer-implemented method, the method further includes determining a probability for each potential target of the individual actor, wherein the predicted target includes the individual potential target having the highest probability.

[0040] In some embodiments of the exemplary computer-implemented method, the target route of the at least one potential target comprises a nominal route defined in the map data.

[0041] In some embodiments of the exemplary computer-implemented method, the at least one potential target for each actor includes multiple potential targets for the actor, and the predicted target is a probability distribution over the multiple potential targets.

[0042] In some embodiments of the exemplary computer-implemented method, step (d) includes processing the plurality of feature embeddings to determine predicted interactions between the individual actor and other actors of the plurality of actors based on the predicted goal.

[0043] In some embodiments of the exemplary computer-implemented method, the predicted interactions are further based on potential goals for the other actors in the environment that intersect with the predicted goals for the individual actor.

[0044] In some embodiments of the exemplary computer-implemented method, the predicted interactions include probability distributions for one or more interaction types between the individual actor and the other actors.

[0045] In some embodiments of the exemplary computer-implemented method, the predicted probability of interaction between the individual actor and the other actor is based on one or more distances between the individual actor and the predicted goal and a shared space associated with the potential goal for the other actor.

[0046] In some embodiments of the exemplary computer-implemented method, the method includes performing one or more message passing rounds to generate a plurality of updated feature embeddings, and the method further includes determining the predicted goals and the continuous trajectories for the individual actors based on the plurality of updated feature embeddings.

[0047] For example, in one aspect, the present disclosure provides an exemplary computing system. The computing system includes one or more processors and one or more non-transitory computer-readable media storing executable instructions to enable the one or more processors to perform operations. The operations include (a) obtaining state data related to a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include (b) determining, based on the state data and the map data, a plurality of potential targets including at least one potential target for each of the plurality of actors, the potential targets including a potential destination location in the environment and a destination route to the potential destination location. The operations include (c) processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to generate a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential targets. The operations include (d) processing the plurality of feature embeddings to determine a predicted goal for each of the plurality of actors and a continuous trajectory for the each of the plurality of actors based on the predicted goal. The actions include (e) initiating movement of the autonomous vehicle based on the predicted goals and the continuous trajectories of the individual actors.

[0048] In some embodiments of the exemplary computing system, the machine learning predictive model includes the graph neural network having individual nodes corresponding to the plurality of actors and the plurality of potential targets.

[0049] In some embodiments of the exemplary computing system, the graph neural network includes one or more actor-goal edges that individually connect each actor node to each goal node.

[0050] In some embodiments of the exemplary computing system, the graph neural network includes one or more goal-to-goal edges that respectively connect at least two goal nodes of the individual nodes corresponding to the plurality of potential goals.

[0051] In some embodiments of the exemplary computing system, the operations include performing one or more rounds of message passing to generate multiple updated feature embeddings.

[0052] In some embodiments of the exemplary computing system, the operations include determining the predicted goals and the continuous trajectories for the individual actors based on the updated feature embeddings.

[0053] In some embodiments of the exemplary computing system, (d) includes processing the plurality of feature embeddings to determine (i) a predicted goal for each actor of the plurality of actors, (ii) a predicted interaction between the each actor and other actors of the plurality of actors based on the predicted goal, and (iii) a continuous trajectory for the each actor based on the predicted goal and the predicted interaction.

[0054] For example, in one aspect, the present disclosure provides an exemplary control system for an autonomous vehicle. The control system includes one or more processors and one or more non-transitory computer-readable media storing executable instructions to enable the one or more processors to perform operations. The operations include (a) acquiring state data related to a plurality of actors in an environment of the autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include (b) determining, based on the state data and the map data, a plurality of potential targets including at least one potential target for each of the plurality of actors, the potential targets including a potential destination location in the environment and a destination route to the potential destination location. The operations include (c) processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to generate a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential targets. The operations include (d) processing the plurality of feature embeddings to determine a predicted target for each of the plurality of actors and a continuous trajectory for the each of the plurality of actors based on the predicted target. The actions include (e) initiating movement of the autonomous vehicle based on the predicted goals and the continuous trajectories of the individual actors.

[0055] Other exemplary aspects of the present disclosure relate to other systems, methods, vehicles, apparatus, tangible non-transitory computer-readable media, and devices for generating trajectories, learning models, and performing other functions described herein. These and other features, aspects, and advantages of various embodiments will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain associated principles. [Brief explanation of the drawings]

[0056] A detailed description of embodiments presented to those skilled in the art is set forth herein with reference to the accompanying drawings.

[0057] [Figure 1] FIG. 1 is a block diagram of an operational scenario according to some embodiments of the present disclosure.

[0058] [Figure 2] FIG. 1 is a block diagram of a system according to some embodiments of the present disclosure.

[0059] [Figure 3a] FIG. 1 illustrates an operating environment according to some embodiments of the present disclosure.

[0060] [Figure 3b] FIG. 1 illustrates a map of an operating environment according to some embodiments of the present disclosure.

[0061] [Figure 3c] FIG. 1 illustrates an operating environment according to some embodiments of the present disclosure.

[0062] [Figure 3d] FIG. 1 illustrates a map of an operating environment according to some embodiments of the present disclosure.

[0063] [Figure 4] FIG. 1 is a block diagram of a system for initiating movement of an autonomous platform according to some embodiments of the present disclosure.

[0064] [Figure 5] 1 is an exemplary environment that can be transformed for input into a predictive system according to some embodiments of the present disclosure.

[0065] [Figure 6] 1 is a graphical representation of a graph neural network (GNN) generated using a machine learning predictive model according to some embodiments of the present disclosure.

[0066] [Figure 7] FIG. 1 is a block diagram of an example system architecture of a forecasting system according to some embodiments of the present disclosure.

[0067] [Figure 8] FIG. 1 is an exemplary diagram of data flow within a machine learning predictive model according to some embodiments of the present disclosure.

[0068] [Figure 9] 1 is an exemplary environment that may be transformed for input into a predictive system according to some embodiments of the present disclosure.

[0069] [Figure 10] FIG. 1 is an example diagram of data flow associated with continuous trajectories within a machine learning predictive model according to some embodiments of the present disclosure.

[0070] [Figure 11] FIG. 1 is an exemplary diagram of generating labeled training data for training a machine learning predictive model according to some embodiments of the present disclosure.

[0071] [Figure 12] FIG. 1 is an exemplary diagram of generating labeled training data for training a machine learning predictive model according to some embodiments of the present disclosure.

[0072] [Figure 13] 1 is a flowchart of an example method for initiating movement of an autonomous vehicle according to some embodiments of the present disclosure.

[0073] [Figure 14] 1 is an exemplary flowchart of a method for processing data with a machine learning predictive model according to some embodiments of the present disclosure.

[0074] [Figure 15] 1 is an exemplary flowchart of a method for processing data with a machine learning predictive model according to some embodiments of the present disclosure.

[0075] [Figure 16] 1 is a flowchart of an example method for training a machine learning predictive model according to some embodiments of the present disclosure.

[0076] [Figure 17] 1 is a flowchart of an example method according to some embodiments of the present disclosure for generating training data.

[0077] [Figure 18] FIG. 1 is a block diagram of a computing system according to some embodiments of the present disclosure for predicting the intention of one or more objects in an environment. DETAILED DESCRIPTION OF THE INVENTION

[0078] For illustrative purposes only, the techniques of this disclosure are described below within the context of autonomous vehicles. As described herein, the techniques of this disclosure are not limited to autonomous vehicles, but can be implemented for or within autonomous robots, other autonomous platforms, and other computing systems.

[0079] Exemplary embodiments of the present disclosure will be described in more detail with reference to Figures 1-17. Figure 1 is a block diagram of an exemplary operational scenario according to some embodiments of the present disclosure. In the exemplary operational scenario, an environment 100 includes an autonomous platform 110 and multiple objects, including a first actor 120, a second actor 130, and a third actor 140. In the exemplary operational scenario, the autonomous platform 110 can move through the environment 100 and interact with objects located within the environment 100 (e.g., the first actor 120, the second actor 130, the third actor 140, etc.). The autonomous platform 110 can be configured to selectively communicate with a remote system 160 via a network 170.

[0080] Environment 100 may be or include an indoor environment (e.g., an environment within one or more facilities, etc.) or an outdoor environment. An indoor environment may be, for example, an environment enclosed by a structure such as a building (e.g., a service warehouse, a maintenance location, a manufacturing facility, etc.). An outdoor environment may be, for example, one or more regions of the outside world, such as one or more rural areas (e.g., having one or more rural travel routes, etc.), one or more urban areas (e.g., having one or more urban travel routes, highways, etc.), one or more suburban areas (e.g., having one or more suburban travel routes, etc.), or other outdoor environments.

[0081] Autonomous platform 110 may be any type of platform configured to operate within environment 100. For example, autonomous platform 110 may be a vehicle configured to perceive and operate autonomously within environment 100. The vehicle may be, for example, a ground-based autonomous vehicle such as an autonomous car, truck, van, etc. Autonomous platform 110 may be an autonomous vehicle that can control, be coupled to, or otherwise associate with tools, attachments, and / or accessories for transporting people or cargo. The autonomous platform may include, for example, an autonomous tractor selectively coupled to a cargo trailer. Additionally or alternatively, autonomous platform 110 may be any other type of vehicle, such as one or more aircraft, water vehicles, space-based vehicles, other ground-based vehicles, etc.

[0082] Autonomous platform 110 may be configured to communicate with remote system 160. For example, remote system 160 may communicate with autonomous platform 110 for assistance (e.g., mobility assistance, situational response assistance, etc.), control (e.g., fleet management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, remote system 160 may provide data that directs the tasks that autonomous platform 110 should perform. For example, as described further herein, remote system 160 may provide data that directs that autonomous platform 110 should perform a trip / service, such as a user transportation trip / service, a delivery trip / service (e.g., of cargo, freight, goods), etc.

[0083] Autonomous platform 110 may communicate with remote system 160 using network 170. Network 170 may facilitate the transmission of signals (e.g., electronic signals, etc.) or data (e.g., computing device data, etc.) and may include any combination of various wired (e.g., twisted pair cable, etc.) or wireless communication mechanisms (e.g., cellular, radio, satellite, microwave, radio frequency, etc.) or any desired network topology (or topologies). For example, network 170 may include a local area network (e.g., an intranet, etc.), a wide area network (e.g., the Internet, etc.), a wireless LAN network (e.g., via WiFi, etc.), a cellular network, a SATCOM network, a VHF network, an HF network, a WiMAX-based network, or any other suitable communication network (or combination thereof) for transmitting data to or from autonomous platform 110.

[0084] For example, as shown in FIG. 1, environment 100 may include one or more objects. The objects may be objects that do not move or are not expected to move (“static objects”), or objects that move or are expected to move (“dynamic objects” or “actors”). In some embodiments, environment 100 may include any number of actors, such as one or more pedestrians, animals, vehicles, etc. The actors may move within the environment along one or more actor trajectories. For example, first actor 120 may move along any one of first actor trajectories 122A-C, second actor 130 may move along any one of second actor trajectories 132, and third actor 140 may move along any one of third actor trajectories 142.

[0085] As described further herein, autonomous platform 110 may utilize its autonomous system to detect these actors (and their movements) and plan its movement to navigate through environment 100 along one or more platform trajectories 112A-C. Autonomous platform 110 may include an on-board computing system 180. On-board computing system 180 may include one or more processors and one or more memory devices. The one or more memory devices may store instructions executable by the one or more processors to enable the one or more processors to perform operations or functions associated with autonomous platform 110, including implementing the autonomous platform's autonomous navigation system.

[0086] 2 is a block diagram of an example autonomous system 200 for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system 200 can be implemented by a computing system of the autonomous platform (e.g., the onboard computing system 180 of the autonomous platform 110). The autonomous system 200 can operate to obtain inputs from sensors 202 or other input devices. In some embodiments, the autonomous system 200 can further obtain platform data 208 (e.g., map data 210) from local or remote storage. The autonomous system 200 can generate control outputs for controlling the autonomous platform based on the sensor data 204, the map data 210, or other data (e.g., via a platform controller 212, etc.). The autonomous system 200 can include different subsystems for performing various autonomous operations. The subsystems can include a localization system 230, a perception system 240, a planning system 250, and a control system 260. The position estimation system 230 may determine the position of the autonomous platform within the environment, the perception system 240 may detect, classify, and track objects and actors within the environment, the planning system 250 may determine the trajectory of the autonomous platform, and the control system 260 may convert the trajectory into vehicle control actions for control of the autonomous platform. The autonomous system 200 may be implemented by one or more on-board computing systems. The subsystems may include one or more processors and one or more memory devices. The one or more memory devices may store instructions executable by the one or more processors to enable the one or more processors to perform operations or functions associated with the subsystem. The computing resources of the autonomous system 200 may be shared among its subsystems, or a subsystem may have a dedicated set of computing resources.

[0087] In some embodiments, autonomous system 200 can be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). Autonomous system 200 can perform various processing techniques on inputs (e.g., sensor data 204, map data 210) to perceive and understand the vehicle's surroundings and generate an appropriate set of control outputs for implementing a vehicle motion plan (e.g., including one or more trajectories) for navigating the vehicle's surroundings (e.g., such as environment 100 of FIG. 1 ). In some embodiments, an autonomous vehicle implementing autonomous system 200 can navigate, operate, maneuver, etc. with minimal or no interaction from a human operator (e.g., a driver, pilot, etc.).

[0088] In some embodiments, the autonomous platform may be configured to operate in multiple operating modes. For example, the autonomous platform may be configured to operate in a fully autonomous (e.g., autonomous driving, etc.) operating mode in which the autonomous platform can be controlled without user input (e.g., capable of driving and navigating without input from a human operator present at or remote from the autonomous vehicle). The autonomous platform may operate in a semi-autonomous operating mode in which the autonomous platform can operate with some input from a human operator present at the autonomous platform (or a human operator remote from the autonomous platform). In some embodiments, the autonomous platform can enter a manual operating mode in which the autonomous platform can be fully controlled by a human operator (e.g., a human driver, etc.) and can be prohibited or deactivated (e.g., temporarily, permanently, etc.) from performing autonomous operation (e.g., autonomous driving, etc.). The autonomous platform may be configured to operate in other modes, such as a parking or power-saving mode (e.g., used during operations such as trip / servicing wait, charging, etc.). In some embodiments, the autonomous platform can implement vehicle operation assistance technologies (e.g., collision mitigation systems, power-assisted steering, etc.) to assist the human operator of the autonomous platform (e.g., while in manual mode, etc.).

[0089] Autonomous system 200 may be located onboard (e.g., on or within) an autonomous platform and may be configured to operate the autonomous platform in a variety of environments. The environments may be actual environments or simulated environments. In some embodiments, one or more simulation computing devices may simulate one or more of sensors 202, sensor data 204, communications interface 206, platform data 208, or platform controller 212 to simulate operation of autonomous system 200.

[0090] In some embodiments, autonomous system 200 may communicate with one or more networks or other systems via communication interface 206. Communication interface 206 may include, for example, a transmitter, a receiver, a port, a controller, an antenna, or any other suitable component capable of facilitating communication and suitable for interfacing with one or more networks (e.g., network 170 of FIG. 1 ). In some embodiments, communication interface 206 may include multiple components (e.g., antennas, transmitters, or receivers, etc.) that enable it to implement and utilize various communication technologies (e.g., multiple-input, multiple-output (MIMO) technologies, etc.).

[0091] In some embodiments, autonomous system 200 may communicate with one or more computing devices remote from the autonomous platform (e.g., remote system 160) via one or more networks (e.g., network 170) using communication interface 206. For example, in some examples, one or more inputs, data, or functions of autonomous system 200 may be supplemented or replaced by a remote system communicating via communication interface 206. For example, in some embodiments, map data 210 may be downloaded to a remote system via a network using communication interface 206. In some examples, one or more of location estimation system 230, perception system 240, planning system 250, or control system 260 may be updated, influenced, guided, communicated with, etc., by a remote system for assistance, maintenance, situational response redefinition, management, etc.

[0092] The sensors 202 may be located onboard the autonomous platform. In some embodiments, the sensors 202 may include one or more types of sensors. For example, one or more sensors may include an image capture device (e.g., a visible spectrum camera, an infrared camera, etc.). Additionally or alternatively, the sensors 202 may include one or more depth capture devices. For example, the sensors 202 may include one or more light detection and ranging (LIDAR) sensors or radio detection and ranging (RADAR) sensors. The sensors 202 may be configured to generate point data describing at least a portion of a 360-degree view of the surrounding environment. The point data may be point cloud data (e.g., three-dimensional LIDAR point cloud data, RADAR point cloud data). In some embodiments, one or more of the sensors 202 for capturing depth information may be fixed to a rotational device to rotate the sensor 202 about an axis. As the sensor 202 rotates about the axis, it may capture data in interval sector packets that describe different portions of the 360-degree view of the surrounding environment of the autonomous platform. In some embodiments, one or more of the sensors 202 for capturing depth information may be a solid-state sensor.

[0093] The sensors 202 may be configured to capture sensor data 204 that represents or relates to at least a portion of the autonomous platform's environment. The sensor data 204 may include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system 200 may obtain input from additional types of sensors, such as inertial measurement units (IMUs), altimeters, inclinometers, odometry devices, position or positioning devices (e.g., GPS, compasses), wheel encoders, or other types of sensors. In some embodiments, the autonomous system 200 may obtain sensor data 204 related to particular components or systems of the autonomous platform. This sensor data 204 may indicate, for example, wheel speed, component temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system 200 may obtain sensor data 204 related to ambient conditions, such as environmental or weather conditions. In some embodiments, the sensor data 204 may include multi-modal sensor data. Multimodal sensor data can be acquired by at least two types of sensors (e.g., sensors in sensors 202) and can be indicative of static objects or actors within the autonomous platform's environment. The multimodal sensor data can include at least two types of sensor data (e.g., camera and LIDAR data). In some embodiments, the autonomous platform can utilize sensor data 204 for sensors that are remote (e.g., off-board) from the autonomous platform. This can include, for example, sensor data 204 captured by other autonomous platforms.

[0094] Autonomous system 200 may obtain map data 210 related to the environment in which the autonomous platform is located, is located in, or will be located. Map data 210 may provide information about the environment or geographic region. For example, map data 210 may provide information regarding various travel paths (e.g., roads, etc.), travel path segments (e.g., road segments, etc.), the identification and location of buildings or other items or objects (e.g., street lights, crosswalks, curbs, etc.), the location and orientation of boundaries or boundary markers (e.g., the location and orientation of traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.), traffic control data (e.g., the location and indication of signs, traffic signals, other traffic control devices), obstacle information (e.g., temporary or permanent blockages, etc.), event data (e.g., road closures / traffic rule changes due to parades, concerts, sporting events, etc.), nominal vehicle path data (e.g., representing an ideal vehicle path, such as along the center of a particular lane), or any other map data that provides information useful to the autonomous platform for understanding its surrounding environment and its relationships. In some embodiments, map data 210 may include high-resolution map information. Additionally or alternatively, map data 210 may include sparse map data (e.g., lane graphs, etc.). In some embodiments, the sensor data 204 can be fused with the map data 210 or used to update the map data 210 in real time.

[0095] Autonomous system 200 may include a position estimation system 230 that may provide the autonomous platform with an understanding of its position and orientation within its environment. In some examples, position estimation system 230 may assist one or more other subsystems of autonomous system 200, for example, by providing a unified local frame of reference for performing perception, planning, or control operations.

[0096] In some embodiments, position estimation system 230 may determine the autonomous platform's current position. The current position may include an absolute position (e.g., based on a georeferenced anchor, etc.) or a relative position (e.g., based on objects in the environment, etc.). Position estimation system 230 may generally include or interface with any device or circuitry for analyzing the position or changes in position of an autonomous platform (e.g., an autonomous ground-based vehicle, etc.). For example, position estimation system 230 may determine the position by using one or more of an inertial sensor (e.g., an inertial measurement unit, etc.), a satellite position estimation system, a radio receiver, networking equipment (e.g., based on an IP address, etc.), triangulation or proximity to network access points or other network components (e.g., cellular towers, WiFi access points, etc.), or other suitable techniques. The autonomous platform's position may be used by various subsystems of autonomous system 200 or provided to a remote computing system (e.g., using communication interface 206).

[0097] In some embodiments, the position estimation system 230 can register the relative positions of elements of the autonomous platform's surrounding environment with positions recorded in the map data 210. For example, the position estimation system 230 can process the sensor data 204 (e.g., LIDAR data, RADAR data, camera data, etc.) to align or otherwise register with a map of the surrounding environment (e.g., from the map data 210) to determine the autonomous platform's location within that environment. Thus, in some embodiments, the autonomous platform can identify its location within the surrounding environment (e.g., across six axes) based on a lookup of the map data 210. In some embodiments, given an initial position, the position estimation system 230 can update the autonomous platform's position by incremental realignment based on recorded or estimated deviations from the initial position. In some embodiments, the position can be registered directly within the map data 210.

[0098] In some embodiments, map data 210 may include a large amount of data subdivided into geographic tiles such that a desired area of ​​a map stored in map data 210 can be reconstructed from one or more tiles. For example, multiple tiles selected from map data 210 may be stitched together by autonomous system 200 based on a location obtained by location estimation system 230 (e.g., multiple tiles selected near a location).

[0099] In some embodiments, position estimation system 230 may determine the position (e.g., relative position or absolute position) of one or more attachments or accessories relative to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and position estimation system 230 may provide the position of one or more points on the cargo platform. For example, the cargo platform may include a trailer or other device towed, otherwise attached, or operated by the autonomous platform, and position estimation system 230 may provide data describing the position (e.g., absolute, relative position, etc.) of the cargo platform as well as the autonomous platform. This information may be obtained by other autonomous navigation systems to assist in the movement of the autonomous platform.

[0100] Autonomous system 200 may include a perception system 240 that enables the autonomous platform to detect, classify, and track objects and actors within the environment. Environmental features or objects recognized within the environment may be within the field of view of sensors 202 or may be predicted to be hidden from sensors 202. This may include objects that do not move or are not predicted to move (static objects), or objects that move or are predicted to move (dynamic objects / actors).

[0101] The perception system 240 may determine one or more states (e.g., current or past states, etc.) of one or more objects within the autonomous platform's surrounding environment. For example, the state may describe an object's current or past location (also referred to as position) (e.g., for a given time, period, etc.), current or past speed / velocity, current or past acceleration, current or past heading, current or past orientation, size / footprint (e.g., represented by a bounding shape, object highlighting, etc.), classification (e.g., pedestrian class vs. vehicle class vs. bicycle class, etc.), associated uncertainty, or an estimate of other state information. In some embodiments, the perception system 240 may determine the state using one or more algorithms or machine learning models configured to identify / classify objects based on input from the sensors 202. The perception system may generate a representation of the environment using different modalities of the sensor data 204, which is processed by one or more algorithms or machine learning models. In some embodiments, the state of one or more identified or unidentified objects may be maintained and updated over time as the autonomous platform continues to recognize and interact with the objects (e.g., launch, yield, etc.) with or around the objects. In this manner, perception system 240 may provide an understanding of the current state of the environment (e.g., state including objects in the environment, etc.) known from a record of previous states of the environment (e.g., state including movement history for objects in the environment). This information may be useful for the autonomous platform in planning movement through the environment.

[0102] Autonomous system 200 may include a planning system 250 that may be configured to determine how the autonomous platform will interact with and move within an environment. Planning system 250 may determine one or more motion plans for the autonomous platform. A motion plan may include one or more trajectories (e.g., motion trajectories) that indicate a path the autonomous platform should follow. A trajectory may be of a particular length or time range. The length or time range may be defined by a computational planning scope of planning system 250. A motion trajectory may be defined by one or more waypoints (including associated coordinates). A waypoint may be a future position of the autonomous platform. Motion plans may be continuously generated, updated, and reviewed by planning system 250.

[0103] The motion planning system 250 may determine a strategy for the autonomous platform. A strategy may be a set of discrete decisions (e.g., yield to actor, back-yield to actor, merge, change lanes) that the autonomous platform makes. The strategy may be selected from multiple potential strategies. The selected strategy may be the lowest-cost strategy determined by one or more cost functions. The cost function may, for example, assess the likelihood of a collision with another actor or object.

[0104] The planning system 250 may determine a preferred trajectory for executing a strategy. For example, the planning system 250 may obtain one or more trajectories for executing one or more strategies. The planning system 250 may evaluate and rank the trajectories or strategies (e.g., using scores, costs, compensations, constraints, etc.). For example, the planning system 250 may use a prediction output representing interactions between the autonomous platform's trajectory and one or more objects (e.g., proximity, intersections, etc.) to inform the evaluation of candidate trajectories or strategies for the autonomous platform. In some embodiments, the planning system 250 may utilize static costs to evaluate the trajectory of the autonomous platform (e.g., "avoid lane boundaries," "minimize jerk," etc.). Additionally or alternatively, the planning system 250 may utilize dynamic costs to evaluate trajectories or strategies for the autonomous platform based on predicted outcomes for a current operating scenario (e.g., predicted trajectories or strategies that lead to interactions between actors, predicted trajectories or strategies that lead to interactions between actors and the autonomous platform, etc.). The planning system 250 can rank the trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system 250 can select a motion plan (and corresponding trajectory) based on a ranking of multiple candidate trajectories. In some embodiments, the planning system 250 can select the highest-ranked candidate or the highest-ranked feasible candidate.

[0105] The planning system 250 can then validate the selected trajectory against one or more constraints before the trajectory is executed by the autonomous platform.

[0106] To aid in motion planning decisions, planning system 250 may be configured to perform predictive functions. Planning system 250 may predict future states of the environment, which may include predicting future states of other actors in the environment. In some embodiments, planning system 250 may predict future states based on current or past states (e.g., developed or maintained by perception system 230). In some embodiments, the future states may be or include predicted trajectories (e.g., positions over time) of objects in the environment, such as other actors. In some embodiments, one or more of the future states may include one or more probabilities associated therewith (e.g., marginal probabilities, conditional probabilities). For example, the one or more probabilities may include one or more probabilities conditioned on strategy or trajectory options available to the autonomous platform. Additionally or alternatively, the probabilities may include probabilities conditioned on trajectory options available to one or more other actors.

[0107] In some embodiments, planning system 250 may perform interactive prediction. Planning system 250 may determine a motion plan for the autonomous platform by understanding how the predicted future state of the environment may be affected by the execution of one or more candidate motion plans. For example, referring again to FIG. 1 , autonomous platform 110 may determine candidate motion plans corresponding to a set of platform trajectories 112A-C corresponding to first actor trajectories 122A-C for first actor 120, trajectory 132 for second actor 130, and trajectory 142 for third actor 140, respectively (e.g., each trajectory correspondence displayed with a matching line style). For example, autonomous platform 110 may predict (e.g., using autonomous system 200) that platform trajectory 112A, which moves autonomous platform 110 more quickly into the area ahead of first actor 120, is associated with first actor 120 reducing its forward speed in accordance with first actor trajectory 122A and more quickly yielding to autonomous platform 110. Additionally or alternatively, the autonomous platform 110 can predict that platform trajectory 112B, which moves the autonomous platform 110 smoothly into the area ahead of the first actor 120, is associated with the first actor 120 slightly reducing its speed and slowly yielding to the autonomous platform 110 in accordance with first actor trajectory 122B. Additionally or alternatively, the autonomous platform 110 can predict that platform trajectory 112C, which maintains parallel alignment with the first actor 120, is associated with the first actor 120 not yielding any distance to the autonomous platform 110 in accordance with first actor trajectory 122C. Based on comparing the predicted scenarios to a desired set of outcomes (e.g., by scoring the scenarios based on cost or compensation), the planning system 250 can select a motion plan (and its associated trajectory) that takes into account the interaction of the autonomous platform with the environment 100. In this manner, for example, the autonomous platform 110 can interleave prediction and motion planning functions.

[0108] To implement the selected motion plan, autonomous system 200 may include control system 260 (e.g., a vehicle control system). Generally, control system 260 may provide an interface between autonomous system 200 and platform controller 212 to implement the strategy and motion plan generated by planning system 250. For example, control system 260 may implement the selected motion plan / trajectory to control the movement of the autonomous platform through the environment by following a selected trajectory (e.g., waypoints included in the trajectory). For example, control system 260 may translate the motion plan into commands (e.g., acceleration control, braking control, steering control, etc.) to the appropriate platform controller 212. For example, control system 260 may translate the selected motion plan into commands to adjust a steering component (e.g., steering angle) by a specific number, apply braking force of a specific size, increase / decrease speed, etc. In some embodiments, control system 260 may communicate with platform controller 212 via a communication channel including, for example, one or more data buses (e.g., a controller area network (CAN) or the like), an on-board diagnostic connector (e.g., OBD-II or the like), or a combination of wired or wireless communication links. Platform controller 212 may send or receive data, messages, signals, etc. to or from autonomous system 200 (or vice versa) via the communication channel.

[0109] Autonomous system 200 may receive an assistance signal from remote assistance system 270 via communication interface 206. Remote assistance system 270 may communicate with autonomous system 200 over a network (e.g., as remote system 160 via network 170). In some embodiments, autonomous system 200 may initiate a communication session with remote assistance system 270. For example, autonomous system 200 may initiate the session based on or in response to a trigger. In some embodiments, the trigger may be a warning, an error signal, a map feature, a request, a location, a traffic condition, a road condition, etc.

[0110] After initiating a session, the autonomous system 200 may provide context data to the remote assistance system 270. The context data may include sensor data 204 and autonomous platform state data. For example, the context data may include a live camera feed from the autonomous platform's camera and the autonomous platform's current speed. An operator (e.g., a human operator) of the remote assistance system 270 can use the context data to select an assistance signal. The assistance signal may provide values ​​or adjustments for various operating parameters or characteristics of the autonomous system 200. For example, the assistance signal may include waypoints (e.g., routing around obstacles, lane changes, etc.), speed or acceleration profiles (e.g., speed limits, etc.), relative movement instructions (e.g., convoy formation, etc.), operating characteristics (e.g., use of auxiliary systems, reduction of energy processing modes, etc.), or other signals to assist the autonomous system 200.

[0111] Autonomous system 200 may use assistance signals input to one or more autonomous subsystems to perform autonomous functions. For example, planning subsystem 250 may receive assistance signals as inputs for generating a motion plan. For example, assistance signals may include constraints for generating a motion plan. Additionally or alternatively, assistance signals may include cost or compensation adjustments that affect the motion plan by planning subsystem 250. Additionally or alternatively, assistance signals may be considered as suggestive inputs that may be considered by autonomous system 200 along with other received data (e.g., sensor inputs, etc.).

[0112] Autonomous system 200 may be platform independent, and control system 260 may provide control instructions to platform controller 212 for a variety of platforms for autonomous travel (e.g., multiple different autonomous platforms equipped with the autonomous control system), which may include a variety of different types of autonomous vehicles (e.g., sedans, vans, SUVs, trucks, electric vehicles, combustion-powered vehicles, etc.) from a variety of different manufacturers / developers operating in a variety of different environments and, in some embodiments, one or more vehicle services.

[0113] For example, referring to FIG. 3a, the operating environment may include a congested environment 300. The autonomous platform may include an autonomous vehicle 310 controlled by the autonomous system 200. In some embodiments, the autonomous vehicle 310 may be configured for maneuverability in a congested environment, for example, with a set wheelbase or other specifications. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo or passengers. In some embodiments, the autonomous vehicle 310 (e.g., a passenger van, shuttle, bus, etc.) may be configured to transport a large number of passengers. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo, for example, large cargo (e.g., a truck, box van, step van, etc.), or small cargo (e.g., food, personal packages, etc.).

[0114] Referring to FIG. 3b, a selected overhead view 302 of dense environment 300 is shown overlaid with an exemplary trip / service between a first location 304 and a second location 306. The exemplary trip / service may be assigned to an autonomous vehicle 320, for example, by a remote computing system. The autonomous vehicle 320 may be, for example, the same type of vehicle as the autonomous vehicle 310. The exemplary trip / service may include transporting passengers or cargo between the first location 304 and the second location 306. In some embodiments, the exemplary trip / service may include traveling to or through one or more intermediate locations, for example, to load or unload passengers or cargo. In some embodiments, the exemplary trip / service may be pre-booked (e.g., for regular service according to a transportation schedule, etc.). In some embodiments, the exemplary trip / service may be on-demand (e.g., by requesting or for performance of a taxi, ride-sharing, ride-hailing, courier service, delivery service, etc.).

[0115] Referring to FIG. 3c, in another example, the operating environment may include an open mobile path environment 330. The autonomous platform may include an autonomous vehicle 350 controlled by the autonomous system 200, which may include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle 350 may be configured for high-load transportation, such as long-distance, high-load transportation (e.g., transporting large amounts of cargo or other goods or passengers). For example, the autonomous vehicle 350 may include one or more cargo platform attachments, such as a trailer 352. While FIG. 3c is shown as a tow-type attachment, in some embodiments, one or more cargo platforms may be integrated into the autonomous vehicle 350 (e.g., mounted on a chassis) (e.g., as in a box van, step van, etc.).

[0116] Referring to FIG. 3d, a selected overhead view of an open travel path environment 330 is shown, including travel path 332, interchange 334, transfer hubs 336 and 338, access travel path 340, and locations 342, 344. In some embodiments, an autonomous vehicle (e.g., autonomous vehicle 310 or autonomous vehicle 350) can be assigned an exemplary trip / service to operate one or more travel paths 332 (optionally connected by interchange 334) to transport cargo between transfer hub 336 and transfer hub 338. For example, in some embodiments, the exemplary trip / service includes a cargo delivery / transportation service, such as a freight delivery / transportation service. The exemplary trip / service can be assigned by a remote computing system. In some embodiments, transfer hub 336 can be a cargo origination point (e.g., a warehouse, wholesale store, facility, etc.), and transfer hub 338 can be a cargo arrival point (e.g., a retail store, etc.). However, in some embodiments, transfer hub 336 may be an intermediate point along the final journey between each origin and destination of the cargo item. For example, the origin of the cargo item may be located at location 342 along access travel path 340. Thus, the cargo item may be transported (e.g., by a human-operated vehicle, autonomous vehicle 310, etc.) to transfer hub 336 for staging. At transfer hub 336, various cargo items may be grouped or staged for long-distance transportation along travel path 332.

[0117] In some embodiments of the exemplary trip / service, a group of staged cargo items may be loaded onto an autonomous vehicle (e.g., autonomous vehicle 350) for transport to one or more other transit hubs, such as transit hub 338. For example, while not shown, it should be understood that open travel path environment 330 may include more transit hubs than transit hubs 336 and 338 and may include more travel paths 332 interconnected to more interchanges 334. A simplified map is presented here for clarity purposes only. In some embodiments, one or more cargo items to be transported to transit hub 338 may be distributed (e.g., by a human-operated vehicle, autonomous vehicle 310, etc.) to one or more local destinations along access travel path 340, for example, to location 344. In some embodiments, the exemplary trip / service may be reserved in advance (e.g., for regular service according to a transportation schedule, etc.). In some embodiments, the exemplary trip / service may be on-demand (e.g., to provide charter passenger transportation or cargo delivery service or upon request for that service).

[0118] The present disclosure relates to predictive techniques that help improve the planning performance of an autonomous platform 110, such as an autonomous vehicle (e.g., autonomous vehicle 310 or 350) that is at least partially controlled using an autonomous system 200.

[0119] 4 is a block diagram of a system 400 configured to initiate movement of an autonomous platform according to some embodiments of the present disclosure. One or more portions of system 400 may be included in and / or associated with a planning system of the autonomous platform (e.g., planning system 250 of FIG. 2 ). System 400 may include a context memory 405. Context memory 405 may include a cache memory that stores context data that may be processed by prediction system 410, strategy enumeration system 415, strategy ranking system 420, trajectory optimization system 425, and plan arbitration system 430 to determine how the autonomous platform interacts with its environment. The context data may include map data (e.g., map data 210 of FIG. 2 ), sensor data (e.g., sensor data 204 of FIG. 2 ), and / or state data representing one or more current, past, and / or predicted future states of actors in the autonomous vehicle's environment.

[0120] The prediction system 410 may include machine learning prediction models that may process context data to generate discrete intent and continuous motion outputs that can be used to perform one or more portions of the motion planning process. The discrete intent and continuous motion outputs may include one or more predicted goals 435, predicted interactions 440, and / or continuous trajectories 445 for each individual actor in the environment.

[0121] The strategy enumeration system 415 may determine multiple feasible strategies for the autonomous platform based on the context data from the context memory 405. The individual strategies may include multiple discrete decisions that the autonomous platform may make within its operating time within the environment. The multiple discrete decisions may include interpretable decisions, such as, for example, yielding to an actor, merging behind an actor, and / or any other interpretable travel path decision. In some embodiments, the strategy enumeration system 415 may determine multiple feasible strategies for the autonomous platform based on predicted goals 435 and / or predicted interactions 440 for actors in the environment.

[0122] The strategy ranking system 420 may process the possible strategies and context data and determine a ranking of one or more possible strategies. The strategy ranking system 420 may determine a ranking of the possible strategies by one or more costs associated with the strategies. In some cases, the strategy ranking system 420 may include one or more machine learning models trained on expert human driving data. The one or more machine learning models may be trained to determine a cost corresponding to the difference between an individual strategy and a strategy of an expert human driver in the same driving scenario. In some embodiments, the strategy ranking system 420 may consider predicted goals 435 and / or predicted interactions 440 for actors in the environment when determining the ranking of the possible strategies.

[0123] The trajectory optimization system 425 can identify an optimal trajectory for each of the identified strategies based on the context data and one or more cost functions. For example, the cost functions can include static cost functions that encode one or more desired driving behaviors, such as avoiding lane boundaries, staying near the center of the lane, avoiding acceleration and / or jerks, avoiding steering jerks, etc. Additionally or alternatively, the cost functions can include dynamic cost functions that can evaluate dynamic constraints. For example, the dynamic cost functions can evaluate predicted goals 435, predicted interactions 440, and / or continuous trajectories 445 for actors in the environment.

[0124] The plan arbitration system 430 can select an optimal trajectory and strategy pair for execution by the autonomous platform. To do so, the plan arbitration system 430 can reject one or more trajectories that result in interference with other actors / objects, lane boundary violations, etc. The plan arbitration system 430 can select an optimal trajectory and strategy pair from among the non-rejected trajectories that optimize (e.g., minimize) a total cost as evaluated by the static and / or dynamic cost functions described herein. In some embodiments, the plan arbitration system 430 can select an optimal trajectory and strategy pair based on predicted goals 435 for actors in the environment.

[0125] The autonomous platform may initiate movement based on the selected optimal trajectory and strategy pair. In this manner, the autonomous platform's movement may be initiated based on the predicted goal 435, predicted interaction 440, and / or continuous trajectory 445 output by the prediction system 410.

[0126] Figure 5 is an example environment 500 that may be modeled by data provided as input to a prediction system to determine predicted goals, predicted interactions, and / or continuous trajectories according to some embodiments of the present disclosure. More specifically, data representing the environment 500 of Figure 5 may be provided as input to an example prediction system 700 of Figure 7 that is trained to encode the data into a graph neural network (GNN) 600, as shown in Figure 6.

[0127] The environment 500 may include multiple actors 510A-B. The actors may be dynamic objects such as, for example, vehicles, pedestrians, and / or any other objects that change position within the environment 500 over time.

[0128] The prediction system 700 of FIG. 7 may obtain state data 702 associated with each of multiple actors in an environment (e.g., actors 510A-B in the environment 500 of FIG. 5). The state data 702 may include physical characteristics (e.g., bounding box, shape, etc.) and / or multiple actor states associated with each actor. The multiple actor states may indicate one or more positions, one or more velocities, and / or one or more headings for each actor at a current time and one or more past time points. For example, the multiple actor states may include one or more past states 525A-B and / or current states 520A-B for each of the multiple actors 510A-B in the environment 500. The current states 520A-B may include a position, velocity, acceleration, heading, etc. at the current time point. The past states 525A-B may include a position, velocity, acceleration, heading, etc. at a time point prior to the current time point. The past time points may fall within one or more past time ranges. The past time range may include one or more seconds, minutes, etc. prior to the current time. In some embodiments, past states 525A-B may include multiple past positions, velocities, accelerations, headings, etc. for each actor within a 5 second time range prior to the current time.

[0129] Additionally or alternatively, the predictive system 700 of FIG. 7 may obtain map data 704 associated with an environment (e.g., map data for the environment 500 of FIG. 5). The map data 704 may indicate multiple lanes of the environment 500 and / or map features for the multiple lanes. For example, the map data 704 may include a lane graph describing the multiple lanes and / or right-of-way information for the multiple lanes. Additionally or alternatively, the map features in the map data 704 may identify lane centerline locations (e.g., nominal waypoints), lane boundary locations, lane boundary types (e.g., physical barriers, solid lines, dashed lines, etc.), speed limits, and / or any other physical or traffic information for the multiple lanes of the environment 500.

[0130] The map data 704 may define a nominal path for each of a plurality of lanes. The nominal path may be defined by a plurality of waypoints along and / or within a critical distance of the centerline of each lane. For example, the plurality of map features may include a plurality of waypoints for each nominal path defined in the map data 704. For each waypoint, the map feature may identify the location, speed limit, and / or any other attributes of the lane at the respective waypoint.

[0131] A portion of the prediction system 700 (e.g., one or more graph generation layers 712 of the encoder 710) may be configured to determine one or more potential goals for the actor from the state data 702 and map data 704 provided as input to the machine learning prediction model 706. The potential goals may indicate the actor's intended destination within the environment 500 and a nominal route to that destination. For example, the potential goals may represent locations within the environment 500 that the actor may occupy in the future. As an example, the potential goal 505A may represent a location within the environment 500 that the first actor 510A would occupy if the first actor 510A decided to make a right turn. The potential goals may coincide with or be determined by lanes available to the actor in the actor's immediate vicinity.

[0132] Each potential goal may include a target path and a target destination. The target destination may include a future destination location for an individual actor in the environment 500. The future destination location may be based on discrete decisions predicted for the individual actor. For example, a future destination location may be identified based on a prediction that the individual actor will turn right, another future destination location may be identified based on a prediction that the individual actor will continue straight, and so on. The target path may include one or more waypoints along a path from the individual actor's current location to the individual future destination location.

[0133] The predictive system 700 may determine potential targets for each actor in the environment 500 based on contextual data about the environment 500 (e.g., available lanes for the vehicle). In some embodiments, the target path for the potential target may include a nominal path defined in the map data 704. For example, the potential target may indicate one of the actor's intended destinations and a nominal path to the intended destination. The map data 704 may include multiple map features associated with each potential target. The map features may identify multiple waypoints of the individual nominal paths for the potential target. In this manner, the potential target may define spatial aspects of the actor's intended future action (e.g., turn right, go straight, etc.) based on the map data 704 corresponding to the environment 500.

[0134] The predictive system 700 may determine a plurality of first potential targets 505A-D for the first actor 510A based on the current state (e.g., location, heading, etc.) of the first actor 510A and the map data. For example, the predictive system 700 may identify each nominal path defined by the map data that includes waypoints within a search radius 530 of the current state 520A of the first actor 510A.

[0135] As an example, the plurality of first potential targets 505A-D for the first actor 510A may include four potential targets 505A-D based on their location within the environment 500. The first target 505A may include a nominal route and destination location if the first actor 510A intends to turn right. The second target 505B may include a nominal route and destination location if the first actor 510A continues straight ahead. The third target 505C may include a nominal route and destination location if the first actor 510A intends to make a left lane change. The fourth target 505D may include a nominal route and destination location if the first actor 510A intends to turn left.

[0136] The prediction system 700 may determine a plurality of second potential targets 515A-D for the second actor 510B within the environment 500 based on the current state (e.g., location, heading, etc.) of the second actor 510B and map data. The plurality of second potential targets 515A-D for the second actor 510B may include four different potential targets 515A-D based on the location of the second actor within the environment 500. The first target 515A may include a nominal route and destination location if the second actor 510B intends to turn left. The second target 515B may include a nominal route and destination location if the second actor 510B continues straight. The third target 515C may include a nominal route and destination location if the second actor 510B intends to change lanes to the right. The fourth target 515D may include a nominal route and destination location if the second actor 510B intends to turn right.

[0137] In some embodiments, the predictive system 700 may determine a goal-free mode (Goal-Free) 535 to indicate that an individual actor (e.g., the first actor 510A) may not intend to make a decision based on any of the identified potential goals 505A-D (e.g., by running off the road, stopping, etc.).

[0138] The prediction system 700 may be configured to predict the probability that each individual actor will follow each goal and / or goalless mode. For example, the prediction system 700 may process the state data 702, the map data 704, and a plurality of potential goals for each actor in the environment 500 as one or more parts of a machine learning predictive model 706 to determine predicted goals for each individual actor of the plurality of actors 510A-B. As an example, the machine learning predictive model 706 may determine the probability of each potential goal for each individual actor. The predicted goals may include each potential goal with the highest probability.

[0139] As described in more detail herein, the machine learning predictive model 706 may also determine at least one predicted interaction between each of the plurality of actors 510A-B and other actors based on the predicted goal and at least one continuous trajectory for each of the actors based on the predicted goal.

[0140] 6 is a graphical representation of a graph neural network (GNN) 600 generated using a machine learning predictive model (e.g., machine learning predictive model 706 of FIG. 7 ) according to some embodiments of the present disclosure. Graph neural network 600 may include multiple nodes and multiple edges. The multiple nodes and multiple edges may be based on the multiple actors and multiple goals of environment 500 of FIG. 5 . The multiple goals and multiple edges represented in GNN 600 of FIG. 6 may be generated within graph generation layer 712 of machine learning predictive model 706 of FIG. 7 .

[0141] For example, the plurality of nodes of GNN 600 may include a plurality of actor nodes 610A-B corresponding individually to each actor of the plurality of actors 510A-B in environment 500. The plurality of actor nodes 610A-B may include at least one actor node for each actor in environment 500. As an example, the plurality of actor nodes 610A-B may include a first actor node 610A corresponding to a first actor 510A in environment 500 and / or a second actor node 610B corresponding to a second actor 510B in environment 500.

[0142] The plurality of nodes of the GNN 600 may further include a plurality of goal nodes corresponding individually to each potential target of a plurality of potential targets identified for the actors 510A-B. The plurality of goal nodes may include at least one goal node for each target identified for each actor in the environment 500. For example, the plurality of goal nodes may include a plurality of first goal nodes 605A-D corresponding to a first actor node 610A (e.g., for the first actor 510A) and a plurality of second goal nodes 615A-D corresponding to a second actor node 610B (e.g., for the second actor 610B).

[0143] The plurality of edges of the GNN 600 may include one or more actor-goal edges connecting individual actor nodes and individual goal nodes. The actor-goal edges may include at least one actor-goal edge between an individual actor and each goal identified for that individual actor in the environment. As an example, the plurality of actor-goal edges may include a plurality of first actor-goal edges 620A-D corresponding to the first actor node 610A and the plurality of first goal nodes 605A-D (e.g., for a plurality of potential goals 505A-D identified for a first actor 510A). Additionally or alternatively, the plurality of actor-goal edges may include a plurality of second actor-goal nodes 625A-D corresponding to the second actor node 610B and the plurality of second goal nodes 615A-D (e.g., for a plurality of potential goals 515A-D identified for a second actor 510B).

[0144] The plurality of edges may further include one or more goal-goal edges 630A-F, each connecting at least two goal nodes of the plurality of goal nodes. The goal-goal edges may indicate a relationship between two goal nodes that are individually associated with at least two actors in the environment. As an example, the goal-goal edges 630A-F may include: (1) a first goal-goal edge 630A indicating a relationship between a first goal node 605A associated with a first actor 510A and a first goal node 615A associated with a second actor 510B; (2) a second goal-goal edge 630B indicating a relationship between a first goal node 615A associated with a second actor 510B and a third goal node 605C associated with the first actor 510A; and (3) a third goal-goal edge 630B indicating a relationship between a first goal node 615A associated with a second actor 510B and a second goal node 605B associated with the first actor 510A. (4) a fourth goal-goal edge 630D indicating a relationship between a fourth goal node 615D associated with the second actor 510B and a fourth goal node 605D associated with the first actor 510A; (5) a fifth goal-goal edge 630E indicating a relationship between the fourth goal node 605D associated with the first actor 510A and a second goal node 615B associated with the second actor 510B; and (6) a sixth goal-goal edge 630F indicating a relationship between the fourth goal node 605D associated with the first actor 510A and a third goal node 615C associated with the second actor 510B.

[0145] The relationship between at least two goal nodes connected by respective goal-goal edges may be based on a potential interaction between the two actors when the two actors move according to the at least two goals represented by the connected goal nodes.

[0146] The nodes and edges of graph neural network 600 may include multiple feature embeddings. The multiple feature embeddings may encode different characteristics based on the corresponding nodes and / or edges of graph neural network 600. For example, the multiple feature embeddings may include multiple actor feature embeddings corresponding to the multiple actor nodes 610A-B, multiple goal feature embeddings corresponding to the multiple goal nodes, multiple actor-goal feature embeddings corresponding to the multiple actor-goal edges, and one or more goal-goal feature embeddings corresponding to one or more goal-goal edges 630A-F.

[0147] Referring again to FIG. 7 , the prediction system 700 can process the state data 702 and map data 704 using an encoder portion 710 of a machine learning prediction model 706. The encoder portion 710 can include multiple encoding layers. The multiple encoding layers can be shared among multiple encoding tasks and can include multiple graph generation layers 712 and multiple message forwarding layers 714. The multiple graph generation layers 712 are for determining nodes and edges in a GNN (e.g., the GNN 600 of FIG. 6 ) according to multiple potential goals determined for actors in the environment and generating latent representations for each node and edge. The latent representations can be in the form of multiple embeddings, including a goal feature embedding, an actor feature embedding, a goal-goal feature embedding, and an actor-goal feature embedding. While FIG. 7 depicts the encoding for all these feature embeddings as occurring within a single encoder portion 710, it should be understood that separate encoder networks or models (e.g., a goal feature model, an actor feature model, a goal-goal feature model, and an actor-goal feature model) can be provided to generate and update different types of embeddings.

[0148] Once the latent representations for nodes and edges are encoded by the graph generation layer 712 of the encoder portion 710, one or more message forwarding layers 714 (e.g., layers similar to the convolutional layers of a CNN) are used to learn updated graph embeddings and generate compact feature representations 720.

[0149] More specifically, one or more message forwarding layers 714 of the machine learning predictive model may perform one or more message forwarding rounds to share information between multiple nodes and multiple edges of the graph neural network to perform scene-level inference. For example, the machine learning predictive model 706 may perform one or more message forwarding rounds via successive message forwarding 714 to generate multiple updated feature embeddings based on information from neighboring feature embeddings. During each message forwarding round, each feature embedding can be updated based on information encoded in each neighboring feature embedding. In other words, message forwarding between nodes / edges of the GNN implemented in the message forwarding layers 714 helps capture dependencies within the graph by having nodes in the graph aggregate messages from neighboring nodes. For example, for a given target node represented by one or more feature embeddings, each message forwarding layer is responsible for updating the feature embedding of the given target node to include data associated with previous representations of the node and data associated with the node's nearest neighbors in the graph (e.g., GNN 600). Each iteration of updating the feature embedding may correspond to one of the multiple message forwarding layers 714. At each iteration, the feature embedding data is shared one hop or step along the graph. Thus, the message forwarding layers 714 may be configured to include multiple layers at least as many as the longest path distance in the graph, where the path distance is defined as the number of discrete steps between nodes in the graph.

[0150] In some embodiments, the number of message forwarding rounds corresponding to the multiple layers in the multiple message forwarding layers 714 may be statically and / or dynamically determined based on one or more characteristics of the environment. For example, the number of message forwarding rounds may be based on the number of actors in the environment, the number of potential goals for each actor in the environment, the number of potential interactions between actors in the environment, etc. In some embodiments, the number of message forwarding rounds may be determined by the number of nodes, edges, and / or feature embeddings of the graph neural network. For example, the number of message forwarding rounds may be determined such that individual feature embeddings can be updated based on information in the feature embedding corresponding to the furthest node and / or edge of the graph neural network.

[0151] The machine learning prediction model may include various components for performing message forwarding. The machine learning prediction model may utilize a target-target edge model configured to update the target-target feature embeddings based on a combination of features of adjacent target nodes. For example, the target-target edge model may be trained to update the target-target feature embeddings 830 based on features communicated between target nodes 805A, 805C.

[0152] The machine learning predictive model can utilize an actor-target edge model for message forwarding. The actor-target edge model can be configured to update the actor-target feature embeddings based on a combination of features of adjacent actor and target nodes. For example, the actor-target model can be trained to update the actor-target feature embeddings 825 based on features transferred between actor node 810A and the embedding of target node 805A based on features transferred between the nodes.

[0153] The machine learning predictive model can utilize a target node model for message forwarding. The target node model can be configured to update embeddings associated with the target node based on features of one or more other target nodes. For example, the target node model can update feature embeddings associated with the target node based on an aggregation of embedding updates of other target-target feature embeddings and actor-target feature embeddings.

[0154] The machine learning predictive model can utilize an actor node model for message forwarding. The actor node model can be configured to update embeddings associated with an actor node based on features of one or more other actor nodes. For example, the actor node model can update actor feature embeddings 850 associated with actor node 810A based on features communicated from other actor nodes.

[0155] The message forwarding layer can be trained with the three decoder heads on the same data as the heads (e.g., in the case of the target probability head, labeled ground truth target paths determined by Mean Cross-Track Distances). In some embodiments, the entire machine learning predictive model can be trained in an end-to-end manner. This training can be performed using the systems and hardware described herein.

[0156] 7, feature representation 720 may correspond to or include any suitable representation for computationally describing the multiple updated feature embeddings, such as, but not limited to, feature vectors. Feature representation 720 may include or otherwise describe the multiple learned feature embeddings, including target feature embeddings, actor feature embeddings, target-target feature embeddings, and actor-target feature embeddings, as described herein.

[0157] The feature representation 720 may be decoded using a decoder 730 to provide the output of the machine learning predictive model 706. The decoder 730 may be a learned model trained using a process similar to that described in connection with FIGS. 16 and 17. The decoder 730 may enable the machine learning predictive model 706 to determine various types of outputs. The feature representation 720 may then be provided as an input to a decoder portion 730 of the machine learning predictive model 706. The decoder portion 730 may include multiple decoding layers. The decoder portion 730 can be trained to make predictions from the node and edge embeddings provided in the feature representation 720. More specifically, the decoder model 730 may be configured to generate predictions corresponding to (i) one or more target probabilities 740, (ii) one or more interaction probabilities 750, and / or (iii) one or more continuous trajectories 760.

[0158] The encoder portion 710 of the machine learning predictive model 706 is configured to generate multiple feature embeddings 720 corresponding to multiple actors and multiple latent goals. For example, the encoder portion 710 of the machine learning predictive model 706 can encode multiple state and / or map features from the state data 702 and map data 704 for individual feature embeddings within the multiple feature embeddings 720. The encoder portion 710 may include a differentiable nonlinear transform configured to convert raw inputs (e.g., state / map data) into latent feature embeddings. In some embodiments, the encoder portion 710 may include multiple two-layer multilayer perceptrons as part of a framework within the graph generation layer 712.

[0159] The encoder portion 710 may process the state data 702 to generate actor feature embeddings for individual actor nodes. The actor feature embeddings may be based on multiple actor states (e.g., current, past states, etc.) associated with individual actors of the multiple actors 510A-B. For example, the actor feature embeddings may encode information related to the current and / or past position, heading, footprint, etc. of each actor in the actor coordinate frame.

[0160] Each actor node may include an individual actor feature embedding. For example, a first actor node 610A may include an actor feature embedding that encodes the first actor's current and / or past location, heading, footprint, etc. Additionally or alternatively, a second actor node 610B may include an actor feature embedding that encodes the second actor's current and / or past location, heading, footprint, etc.

[0161] The encoder portion 710 may process the map data 704 to generate target feature embeddings for individual target nodes. The target feature embeddings may be generated by processing a plurality of map features associated with the destination locations of the individual targets. For example, the target feature embeddings may encode information related to the destination locations of the individual targets within the environment.

[0162] Each goal node may include an individual goal feature embedding. For example, a first goal node 605A for a first actor 510A may include a goal feature embedding related to the location of a goal destination for the first goal 505A. A second goal node 605B for the first actor 510A may include a goal feature embedding related to the location of a goal destination for the second goal 505B. A first goal node 615A for a second actor 510B may include a goal feature embedding related to the location of a goal destination for the first goal 515A. A second goal node 615B for the second actor 510B may include a goal feature embedding related to the location of a goal destination for the second goal 515B, etc.

[0163] The encoder portion 710 may process the state data 702 and map data 704 to generate actor-goal feature embeddings for each actor-goal edge. The actor-goal feature embeddings may encode information related to multiple actor states for each goal. For example, the multiple actor states for a first goal may be multiple actor states in the path tangent frame.

[0164] Each actor-goal edge may include an individual actor-goal feature embedding. For example, the first actor-goal edge 620A may include an individual actor-goal feature embedding that encodes current and / or past actor state information associated with the first actor 510A corresponding to the first actor node 610A relative to the first goal 505A corresponding to the first goal node 605A, and the first actor-goal edge 625A may include an individual actor-goal feature embedding that encodes current and / or past actor state information associated with the first actor 510B corresponding to the second actor node 610B relative to the first goal 515A corresponding to the first goal node 615A.

[0165] The encoder portion 710 may process the state data 702 and the map data 704 to generate goal-to-goal feature embeddings for individual goal-to-goal edges. The individual goal-to-goal feature embeddings may, for example, encode information related to multiple actor states for an individual actor (e.g., a first actor 510A) relative to another actor (e.g., a second actor 510B) of the plurality of actors 510A-B. For example, the individual goal-to-goal feature embeddings may encode state information for an individual actor within the frame of another actor.

[0166] Additionally, the goal-goal feature embedding may further encode one or more distances between the actor and the shared space associated with at least two goals connected by the respective goal-goal edges. For example, the distances may include a first distance between the respective actor and the shared space and a first distance between the other actor and the shared space.

[0167] Each goal-goal edge may include a respective goal-goal feature embedding. As an example, a first goal-goal edge 630A between a first goal node 605A associated with a first actor node 610A and a first goal node 615A associated with a second actor node 610B may include a goal-goal feature embedding that encodes state information for the first actor 510A corresponding to the first actor node 610A relative to the second actor 510B corresponding to the second actor node 610B. The first goal-goal feature embedding may also encode the distance between the first actor 510A and the shared space 710 and the distance between the second actor 510B and the shared space 710.

[0168] As previously described, the machine learning predictive model 706 may process the multiple feature embeddings provided in the feature representation 720 using the decoder portion 730 to determine predicted goals 740, predicted interactions 750, and / or continuous trajectories 760 for each actor in the environment.

[0169] In some embodiments, the techniques of this disclosure may evaluate a scene and make predictions related to it by representing an autonomous vehicle that includes (or is running) a graph neural network using its onboard computing system as a node in the graph neural network. In this way, the autonomous vehicle's computing system can evaluate the autonomous vehicle as an actor in the scene.

[0170] To this end, the computing system may obtain state data associated with the autonomous vehicle and map data representing multiple lanes of the environment. The state data associated with the autonomous vehicle may be generated using one or more of the vehicle's on-board systems. For example, the state data may be associated with the autonomous vehicle's position estimation system (e.g., position estimation system 230 of FIG. 2) or an inertial measurement unit (IMU). For example, the position information encoded in the coded state may be generated by the position estimation system. Additionally or alternatively, information such as speed, acceleration, and heading encoded in the state data may be generated by the IMU.

[0171] As also described herein, the computing system may determine at least one potential destination for the autonomous vehicle based on the state data and the map data. The potential destination may include a potential destination location within the environment and a target path to the potential destination location for the autonomous vehicle.

[0172] The computing system may use a machine learning predictive model to process the state data, map data, and at least one potential goal to determine (i) predicted goals for the autonomous vehicle, (ii) predicted interactions between the autonomous vehicle and other actors based on the predicted goals, and (iii) a continuous trajectory for the autonomous vehicle based on the predicted goals. This may include, for example, representing the autonomous vehicle as a node in a graph neural network. In this manner, the autonomous vehicle may utilize the techniques described herein to predict and plan movements regarding potential interactions between the autonomous vehicle and other actors in the vehicle's environment.

[0173] FIG. 8 is an exemplary diagram of data flow within a system 800 according to some embodiments of the present disclosure. The system 800 may correspond to the prediction system 700 of FIG. 7, which can be used to generate a graph neural network including multiple actor nodes 810A-C, multiple goal nodes 805A-C, multiple actor-goal edges 815A-C, and multiple goal-goal edges 820A-C. Each of the multiple nodes and edges may be associated with a separate feature embedding. For example, the actor-goal edge 815A may include an actor-goal feature embedding 825, the goal-goal edge 820A may include a goal-goal feature embedding 830, and the actor node 810A may include an actor feature embedding 850. These feature embeddings 825, 830, and 850 may be included as part of a feature representation communicated to a decoder 860. The decoder 860 may be trained to receive the feature representations and generate multiple predictions corresponding to a target probability 835, a continuous trajectory 840, and an interaction probability 845. The decoder 860 may correspond to the decoder 730 as described with reference to FIG.

[0174] The machine learning predictive model 706 may, for example, determine target probabilities 835, interaction probabilities 845, and / or continuous trajectories 840 for individual actors in the environment based on multiple updated feature embeddings.

[0175] The machine learning prediction model 706 can predict a target probability 835 for each actor. The target probability 835 can indicate the likelihood that a particular actor will follow each available potential target path or will not follow any of the target paths. The target probability 835 can take the form of a probability distribution over potential target paths, including the case where there is no target path. To determine the target probability 835, the machine learning prediction model can process the actor feature embeddings and the actor-goal feature embeddings. For example, the machine learning prediction model 706 can process at least one of the actor feature embeddings 850 or the actor-goal feature embeddings 825 to determine the target probability 835 for an individual actor corresponding to actor node 810A. In this process, the machine learning prediction model 706 can determine at least one of a goal-based score or a no-goal score 855 for the individual actor. The goal-based score can cause the machine learning prediction model 706 to determine the probability that an individual actor will move according to each potential goal and output a probabilistic value associated with such movement. The no-target score 855 may include a probabilistic target-value indicating the probability that the individual actor will not act in accordance with any identified potential targets. The machine learning predictive model may use the target-based score and / or the no-target score 855 to determine a target probability 835 for the individual actor associated with the actor node 810A. For example, the target probability 835 may be based on the probability associated with each potential target identified for the individual actor and / or the probability that the actor will not act in accordance with any of the identified potential targets. For example, the target probability 835 may include the individual potential target with the highest probability.

[0176] The machine learning prediction model 706 may determine an interaction probability 845 for each pair of overlapping goal paths for different actors. The interaction probability 845 may indicate the likelihood that one actor will yield to, counter-yield to, or ignore the other actor, under the assumption that the actors follow the overlapping goal paths. The interaction probability 845 for a particular pair of overlapping goal paths may be expressed as a probability distribution for such interaction types. For example, the machine learning prediction model 706 may process the goal-goal feature embeddings 830 to determine the interaction probability 845 for overlapping goal paths between the actor corresponding to actor node 810A and another actor corresponding to actor node 810C based on the predicted goals corresponding to goal nodes 805A and 805C.

[0177] The machine learning predictive model 706 may determine a continuous trajectory 840 for each individual actor based on the multiple feature embeddings and the predicted goal for each individual actor. As an example, the machine learning predictive model 706 may determine a continuous trajectory 840 for an individual actor associated with actor node 810A based on the predicted goal 835 and the actor-goal feature embeddings 825. The continuous trajectory 840 may include a predicted future trajectory for the individual actor conditional on a particular goal.

[0178] 9 is an example diagram 900 including continuous trajectories that can be converted to be input into a machine learning prediction system according to some embodiments of the present disclosure. The example diagram 900 includes an environment 500 having a first actor 510A and a second actor 510B. The example diagram 900 includes four continuous trajectories 905A-D conditioned according to each potential goal identified for the first actor 510A. Each continuous trajectory 905A-D may include multiple waypoints representing potential future locations of the first actor 510A if the first actor 510A decides to move according to the identified goal.

[0179] The waypoints of each continuous trajectory 905A-D may be conditioned according to an individual target 505A-D of FIG. 5. For example, continuous trajectory 905A may include multiple waypoints conditioned according to the target path of potential target 505A. Continuous trajectory 905B may include multiple waypoints conditioned according to the target path of potential target 505B. Continuous trajectory 905C may include multiple waypoints conditioned according to the target path of potential target 505C. Trajectory 905D may include multiple waypoints conditioned according to the target path of potential target 505D.

[0180] The continuous trajectory may also include a no-target trajectory 915. The no-target trajectory may indicate a probability that the first actor 510A will not move according to any of the potential trajectories 905A-D. The no-target trajectory 915 may have a null value. Additionally or alternatively, the continuous trajectory may include one or more waypoints that are conditioned according to abnormal behavior (e.g., pulling over) that may cause the first actor 510A to not follow an identified potential target.

[0181] Returning to FIG. 8 , continuous trajectory 840 may be conditioned according to goal probabilities 835 and refined based on information encoded by actor-goal feature embeddings 825. For example, continuous trajectory 840 may include multiple waypoints conditioned according to a goal path associated with goal probabilities 835. The location and / or timing of the waypoints may be offset from the goal path based on state data and / or map data encoded by actor-goal embeddings 825. In some embodiments, continuous trajectory 840 may be generated based on updated data encoded by updated actor-goal embeddings. In this manner, continuous trajectory 840 may be generated based on information aggregated across each of multiple feature embeddings corresponding to multiple nodes and edges of a graph neural network.

[0182] The machine learning predictive model may output target probabilities 835 for each actor in the environment and continuous trajectories 840 conditioned according to the target probabilities 835. The actors may include autonomous platforms operating in the environment. In some embodiments, the machine learning predictive model may condition the target probabilities 835 and / or the continuous trajectories 840 based on known behavior of the autonomous platforms.

[0183] 10 is a diagram 1000 of data flow within a system for predicting actor intent conditioned according to at least one known decision according to some embodiments of the present disclosure. The exemplary diagram 1000 includes a graph neural network generated by the disclosed techniques. The graph neural network may include multiple actor nodes 810A-C, multiple goal nodes 805A-C, multiple actor-goal edges 815A-C, and / or multiple goal-goal edges 820A-C of the graph neural network of FIG. 8.

[0184] The machine learning predictive model 706 may receive vehicle motion data 1010 related to the autonomous platform. The vehicle motion data 1010 may indicate potential motions of the autonomous platform. For example, the vehicle motion data 1010 may identify discrete motion strategies for the autonomous platform. The discrete motion strategies may indicate interpretable driving decisions, such as, for example, going straight, turning, changing lanes, merging into traffic, stopping, etc. In some embodiments, the discrete motion strategies may correspond to specific potential goals for at least one actor (e.g., the autonomous platform) in the environment.

[0185] The machine learning predictive model 706 may generate conditional target probabilities 1055, conditional interaction probabilities 1060, and / or conditional continuous trajectories 1065 for individual actors conditioned according to the vehicle movement data 1010 by integrating information related to the vehicle movement data 1010 into the latent space of the graph neural network.

[0186] For example, the machine learning prediction model 706 may predict marginal target probabilities, marginal interaction probabilities, and marginal continuous trajectories after at least one round of message forwarding between a plurality of actor nodes 810A-C, a plurality of target nodes 805A-C, a plurality of actor-target edges 815A-C, and / or a plurality of target-target edges 820A-C, and generate a plurality of updated feature embeddings for a plurality of nodes and / or edges of the graph neural network.

[0187] The machine learning predictive model 706 may generate conditional feature embeddings 1045 based on the potential movements of the autonomous vehicle. For example, the machine learning predictive model 706 can convert the vehicle movement data 1010 into the conditional feature embeddings 1045. For example, the conditional feature embeddings 1045 may encode state data, map data, and / or any other data related to the vehicle movement data 1010. The machine learning predictive model may concatenate the conditional feature embeddings 1045 with at least one of the multiple updated feature embeddings. For example, the machine learning predictive model may concatenate the conditional feature embeddings 1045 with the corresponding updated goal-target feature embeddings 830. The machine learning predictive model may perform one or more rounds of message forwarding with the concatenated feature embeddings to generate multiple conditional feature embeddings that take into account future movements of the autonomous platform.

[0188] The machine learning predictive model 706 may process the multiple conditional feature embeddings (e.g., using a decoder 1050) to determine conditional goal probabilities 1055, conditional interaction probabilities 1060, and / or conditional continuous trajectories 1065 for individual actors that are conditioned according to the future movements of the autonomous platform. Similar to the decoders described previously herein, the decoder 1050 may be a learned model that is trained using a process similar to that described herein.

[0189] In some embodiments, a graph neural network may include a typed graph neural network model that includes a separate goal-target edge encoder for each interaction type (e.g., yield, reverse yield, ignore, ignorance, etc.) of peripheral predictive interactions. For example, a typed graph neural network model may include a separate yield target-target encoder for processing goal-target feature embeddings associated with peripheral predictive interactions of yield category distributions. A typed graph neural network model may include another reverse yield target-target encoder for processing goal-target feature embeddings associated with peripheral predictive interactions of reverse yield category distributions. A typed graph neural network model may include a separate ignore target-target encoder for processing goal-target feature embeddings unrelated to peripheral predictive interactions. In some embodiments, a typed graph neural network model may include a "blind" target-target encoder for processing goal-target feature embeddings associated with peripheral predictive interactions whose category distributions are unknown or known within a certain confidence threshold. In this way, each target-target encoder can learn different weights to interpret embeddings corresponding to different interaction types.

[0190] The machine learning predictive model 706 can be trained using one or more machine learning techniques. For example, the machine learning predictive model 706 can be trained using one or more supervised, unsupervised, reinforcement, etc. learning techniques. For example, in some embodiments, the machine learning predictive model 706 can be trained using one or more supervised training techniques based on labeled training data. For example, at least a portion of the labeled training data can be input to the machine learning predictive model 706, and the machine learning predictive model 706 can be trained through back-propagation of errors based on the labels.

[0191] The labeled training data may include data representing multiple real environments over time. For example, the data may include map data related to the multiple real environments. Additionally or alternatively, the data may include sensor data (e.g., image data, LIDAR data, etc.) related to the multiple real environments previously recorded by one or more sensors of the autonomous platform. For example, the sensor data may describe one or more traffic sequences representative of the training environment and / or one or more actors within the training environment over a time range.

[0192] The training data may include multiple training labels, which may identify multiple goals and / or multiple interactions within a previously recorded training environment.

[0193] 11 is a diagram of a labeling technique for generating target labels for a labeling environment 1100 according to some embodiments of the present disclosure. The training data may include data related to the labeling environment 1100 over a time range. First, the labeling environment 1100 may include at least one actor 1110 and multiple potential targets 1105A-D for the actor 1110. Additionally or alternatively, the training data may represent a recorded trajectory 1115 for the actor 1110 over a time range. The actor 1110 may be an actual actor (as shown in recorded log data) or a simulated actor. Thus, the acquired training data may be log data of an actual actor or a simulation of an actor.

[0194] The labeling environment 1100 may be processed to generate a plurality of true and / or false labels for each of the potential targets 1105A-D based on the recorded trajectory 1115 for the actor 1110. To this end, the labeling computing system may determine an average traversal distance from the recorded trajectory 1115 to each potential target 1105A-D. For example, the average traversal distance may include an average distance from each individual waypoint of the recorded trajectory 1115 to each waypoint of the target path of each potential target.

[0195] For example, the labeling computing system may determine multiple waypoints for the recorded trajectory 1115. Each of the multiple waypoints may include a respective position of the actor 1110 at a respective time. The labeling computing system may determine one or more first distances between a first waypoint of the recorded trajectory 1115 and a respective first waypoint for each potential target 1105A-D. The first waypoint may correspond to a first time. The labeling computing system may determine a different distance between a next waypoint of the recorded trajectory 1115 and a next respective waypoint for each potential target 1105A-D. For example, the next waypoint may correspond to a second time following the first time. This process may be repeated for each of the multiple waypoints of the recorded trajectory 1115. The average traversal distance between the recorded trajectory 1115 and each potential target may include an average distance between each waypoint of the recorded trajectory 1115 and a corresponding waypoint for each potential target.

[0196] The labeling computing system can compare the average traversal distance between the recorded trajectory 1115 and each potential goal 1105A-D with a threshold distance. If the average traversal distance between the recorded trajectory 1115 and a particular goal 1105A is less than the threshold distance, the labeling computing system can label the particular goal 1105A as a true goal. If the average traversal distance between the recorded trajectory 1115 and the other goals 1105B-D is greater than the threshold distance, the labeling computing system can label the other goals 1105B-D as false goals. A particular goal 1105A labeled as a true goal can identify a potential goal in the environment 1100 through which the actor 1110 attempts to move.

[0197] The critical distance may be any distance. In some embodiments, the threshold distance may be determined based on the labeling environment. For example, the threshold distance may be based on map data corresponding to the labeling environment. For example, the threshold distance may be half the width of an individual lane corresponding to the potential targets 1105A-D.

[0198] In some embodiments, if the recorded trajectory 1115 of the actor 1110 is close to the target path of the potential target for at least 50% of the observed future (e.g., the range of time covered by the training sequence), the potential target may be labeled as true. For example, in the labeling environment 1100, the actor 1110 may follow the target path of a particular target 1105A to make a right turn. As a result, the particular target 1105A for a right turn may be labeled as true, and the other potential targets may be labeled as false.

[0199] The machine learning predictive model 706 may include data related to the labeled environment 110 and may be used to train at least a portion of the machine learning predictive model 706. For example, the labeled targets (e.g., targets labeled as true or false) may be used to train a portion of the machine learning predictive model 706 (e.g., the decoder portion 730) that generates target probabilities. The machine learning predictive model 706 may be trained using training data based on data related to the labeled environment, for example, as described with reference to FIGS. 15 and 16.

[0200] The labeled training data may be generated for training a machine learning prediction model 706 to predict interaction types. FIG. 12 is an exemplary diagram of a potential interaction 700 according to some embodiments of the present disclosure. The potential interaction 700 may be based on a plurality of first potential targets 505A-D for a first actor 510A and a plurality of second potential targets 515A-D for a second actor 510B. The potential interaction 700 may define the temporal aspects of the intentions of each actor. For example, the potential interaction 700 between actors 510A-B may describe how each actor behaves in relation to one another. In this example, the first and second actors 510A-B may be actual actors (as shown in the recorded log data) or simulated actors.

[0201] The interaction 700 can be identified based on a shared space 710 between a first actor 510A and a second actor 510B. The shared space 710 can include a region of space that both the first actor 510A and the second actor 510B attempt to occupy according to their respective potential goals. For example, the shared space 710 can be occupied by the first actor 510A if the first actor 510A decides to turn left according to potential goal 515D. The shared space can also be occupied by the second actor 510B if the second actor 510B decides to go straight according to potential goal 515B.

[0202] The training data may be labeled as discrete interaction types for the potential interactions 700. The discrete interaction types may include a yield or anti-yield interaction type depending on which actor is predicted to reach the shared space 710 first. For example, the actor that reaches the shared space 710 first may be labeled as an anti-yield interaction type, and the actor that reaches the shared space 710 second may be labeled as a yield interaction type. The timing of each actor's arrival at the shared space 710 may be based on state data (e.g., speed, location, etc.) of the first actor 510A and the second actor 510B and / or map data of the environment 500 (e.g., right-of-way, traffic signal status, etc.). A third interaction category may include an ignore category that can be designated when there is no shared space between two potential targets (e.g., targets 505A and 515D) of the first actor 510A and the second actor 510B.

[0203] The labeled training data of Figure 12 can be used to train a machine learning predictive model (e.g., following the processes of Figures 15 and 16), allowing, for example, the predictive system to utilize discrete interaction categories (e.g., counter-yield, yield, ignore) to describe the mutual behavior of first and second actors 510A-B and their potential goals.

[0204] FIG. 13 is a flowchart of a method 1200 according to some embodiments of the present disclosure for initiating movement of an autonomous vehicle. Method 1200 may be performed by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., an autonomous platform, an autonomous system, a prediction system, etc.). Individual portions of method 1200 may be performed by any (or any combination of) one or more computing devices. Additionally, one or more portions of method 1200 may be implemented as an algorithm on hardware components of devices described herein (e.g., FIGS. 1-2, 4, 7, 18, etc.) for performing actor intent prediction as described herein. FIG. 13 depicts elements performed in a particular order for purposes of illustration and discussion. Those skilled in the art will appreciate that, using the disclosure provided herein, elements of any of the methods described herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. FIG. 13 is described with reference to elements / terminology described in connection with other systems and figures for illustrative and illustrative purposes only and is not intended to be limiting. One or more portions of method 1200 may additionally or alternatively be performed by other systems.

[0205] In step 1205, the method 1200 includes obtaining state data associated with a plurality of actors in an environment of the autonomous vehicle and map data representing a plurality of lanes in the environment.

[0206] The state data may include multiple actor states associated with each actor, which may indicate one or more positions, one or more velocities, and / or one or more headings for each actor at the current time and one or more past times.

[0207] At step 1210, method 1200 includes determining, based on the state data and the map data, a plurality of potential goals, including at least one potential goal for each of the plurality of actors. The potential goals may include potential destination locations within the environment and destination routes to the potential destination locations. For example, the potential goals may include future locations defined by the map data.

[0208] In some embodiments, the target route of the potential target may include a nominal route defined in the map data. For example, the map data may include a plurality of map features associated with the potential target. The plurality of map features may include a plurality of waypoints for the nominal route defined in the map data.

[0209] In step 1215, method 1200 includes processing the state data, map data, and multiple potential targets with a machine learning predictive model to determine (i) predicted targets for each of the multiple actors (e.g., expressed in terms of target probabilities), (ii) predicted interactions between each of the multiple actors and other actors based on the predicted targets (e.g., expressed in terms of interaction probabilities), and (iii) continuous trajectories for each of the multiple actors based on the predicted targets.

[0210] In some embodiments, the machine learning predictive model may determine the probability of each potential target for each individual actor, and the predicted target may be the individual potential target with the highest probability.

[0211] The predicted interactions may be based on the predicted goals for each actor in the environment and potential goals for other actors that intersect. The predicted interactions represent the probability of predicted interactions between each actor and other actors. For example, the predicted interactions may be a probability distribution for one or more interaction types (e.g., yielding, counter-yielding, ignoring, ignorance). The probability distribution may be based on one or more distances between the each actor and a shared space associated with the predicted goal and the potential goals of the other actors.

[0212] At step 1220, method 1200 includes initiating autonomous vehicle movement based on the predicted goals, predicted interactions, and / or continuous trajectories for the individual actors. For example, the computing system may initiate autonomous vehicle movement based on the predicted goals, predicted interactions, and / or continuous trajectories for the individual actors.

[0213] FIG. 14 is a flowchart of a method 1300 for processing data with a machine learning predictive model according to some embodiments of the present disclosure. Method 1300 may be performed by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., autonomous platforms, autonomous systems, predictive systems, etc.). Each individual portion of method 1300 may be performed by any (or any combination of) one or more computing devices. Also, one or more portions of method 1300 may be implemented as an algorithm in the hardware components of the devices described herein (e.g., FIGS. 1-2, 4, 7, 18, etc.) for processing information with the machine learning intent models described herein. FIG. 14 depicts elements performed in a particular order for purposes of illustration and discussion. Those skilled in the art will appreciate that, using the disclosure provided herein, elements of any of the methods described herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. FIG. 14 is described with reference to elements / terminology described in connection with other systems and figures for illustrative and illustrative purposes and is not intended to be limiting. One or more portions of the method 1300 may additionally or alternatively be performed by other systems.

[0214] Method 1300 may include sub-operations of operation 1215 of FIG. 13 , where method 1200 includes determining predicted targets, predicted interactions, and continuous trajectories by processing state data, map data, and a plurality of potential targets with a machine learning predictive model.

[0215] In step 1305, the method 1300 includes processing the state data to generate an actor feature embedding based on a plurality of actor states associated with a first actor of the plurality of actors.

[0216] At step 1310, the method 1300 includes processing the map data to generate a target feature embedding based on a first target of the plurality of targets. The target feature embedding may encode a plurality of map features associated with the first potential target. For example, the plurality of map features may include map features (e.g., locations, rights-of-way, etc.) corresponding to each of a plurality of waypoints of a target path to the potential target.

[0217] In step 1315, the method 1300 includes processing the state data and map data to generate an actor-goal feature embedding based on the plurality of actor states for the first goal path.

[0218] In step 1320, the method 1300 includes processing the state data and the map data to generate a goal-goal feature embedding based on the plurality of actor states for a second actor of the plurality of actors.

[0219] In step 1325, the method 1300 includes performing one or more rounds of message passing to generate multiple updated feature embeddings.

[0220] As an example, each feature embedding may be associated with a node and / or edge of a graph neural network. For example, the machine learning predictive model may include a graph neural network including multiple nodes and multiple edges. The multiple nodes may include (i) multiple actor nodes corresponding to each actor of the multiple actors, and (ii) multiple goal nodes corresponding to each potential goal of the multiple potential goals. The multiple edges may include (i) one or more actor-goal edges connecting each actor node to each goal node, and (ii) one or more goal-goal edges connecting at least two goal nodes of the multiple goal nodes. During each message forwarding round, each node and edge of the graph neural network may share information with adjacent nodes and / or edges. As described herein, the machine learning predictive model may utilize learned models / functions (e.g., a goal-goal edge model, an actor-goal edge model, a goal node model) to perform message forwarding that allows feature information to be shared between nodes / edges.

[0221] In step 1330, the method 1300 includes processing at least one of the actor feature embeddings or the actor-goal feature embeddings to determine a predicted goal for the first actor.

[0222] In step 1335, the method 1300 includes processing the goal-goal feature embeddings to determine interactions between the first actor and other actors based on the predicted goals.

[0223] Additionally or alternatively, multiple feature embeddings can be processed in a conditional machine learning intent prediction system according to example method 1400.

[0224] FIG. 15 is a flowchart of another exemplary method 1400 for processing data with a machine learning predictive model according to some embodiments of the present disclosure. Method 1400 may be performed by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual portion of method 1400 may be performed by any (or any combination of) one or more computing devices. Also, one or more portions of method 1400 may be implemented as an algorithm in the hardware components of the devices described herein (e.g., FIGS. 1-2, 4, 7, 18, etc.) for processing information with the machine learning intent models described herein. FIG. 15 depicts elements performed in a particular order for purposes of illustration and discussion. Those skilled in the art will appreciate that, using the disclosure provided herein, elements of any of the methods described herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. FIG. 15 is described with reference to elements / terminology described in connection with other systems and figures for illustrative and illustrative purposes and is not intended to be limiting. One or more portions of method 1400 may additionally or alternatively be performed by other systems.

[0225] Method 1400 may include sub-operations of operation 1215 of Figure 13, where method 1200 includes determining predicted goals, predicted interactions, and continuous trajectories by processing state data, map data, and multiple potential goals with a machine learning predictive model. For example, method 1400 may include operations that may be performed simultaneously with operations 1305-1325 of Figure 14, where method 1300 includes performing one or more message forwarding rounds to generate multiple updated feature embeddings. Method 1400 may include operations that may be performed before operation 1220 of Figure 13, where method 1200 includes initiating autonomous vehicle movement based on predicted goals, predicted interactions, and / or continuous trajectories for individual actors.

[0226] At step 1405, method 1400 includes receiving vehicle motion data for the autonomous vehicle. The vehicle motion data may indicate potential movements of the autonomous vehicle. For example, the vehicle motion data may indicate potential targets selected for execution by the autonomous vehicle.

[0227] In step 1410, the method 1400 includes generating conditional feature embeddings based on the intended movement of the autonomous vehicle.

[0228] In step 1415, the method 1400 includes concatenating the conditional feature embedding with at least one of the plurality of feature embeddings.

[0229] At step 1420, method 1400 includes determining predicted goals, predicted interactions, and continuous trajectories for individual actors by processing the plurality of feature embeddings with a machine learning predictive model. For example, the computing system may process the plurality of feature embeddings with a machine learning conditional predictive model to determine predicted goals, predicted interactions, and continuous trajectories for individual actors.

[0230] FIG. 16 is a flowchart of an exemplary method 1500 for training a machine learning predictive model according to some embodiments of the present disclosure. Method 1500 may be performed by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual portion of method 1500 may be performed by any (or any combination of) one or more computing devices. Also, one or more portions of method 1500 may be implemented as an algorithm in the hardware components of the devices described herein (e.g., FIGS. 1-2, 4, 7, 18, etc.) for training the machine learning intent model described herein. FIG. 16 depicts elements performed in a particular order for purposes of illustration and discussion. Those skilled in the art will appreciate that, using the disclosure provided herein, elements of any of the methods described herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. FIG. 16 is described with reference to elements / terminology described in connection with other systems and figures for illustrative and illustrative purposes and is not intended to be limiting. One or more portions of method 1500 may additionally or alternatively be performed by other systems.

[0231] In step 1505, method 1500 includes generating or acquiring training data. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) may generate the training data, or the training data may be acquired from a real-world log. The training data may include labeled training data, such as data associated with a labeling environment described herein. The labeled training data may identify multiple actors, multiple potential goals for each actor, and whether each actor decided to act in accordance with each individual goal. In some embodiments, the training data may include a previously recorded training environment.

[0232] At step 1510, method 1500 includes inputting at least a portion of the training data into a machine learning predictive model to determine a training prediction goal. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) can input at least a portion of the training data into a machine learning predictive model to determine a training prediction goal.

[0233] At step 1515, method 1500 includes determining a loss for the machine learning predictive model based on the training prediction goal and the plurality of target labels. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) may determine a loss for the machine learning predictive model based on the training prediction goal and the plurality of target labels.

[0234] At step 1520, method 1500 includes updating one or more parameters of the machine learning predictive model based on the loss. For example, the computing system can update one or more parameters of the machine learning predictive model based on the loss.

[0235] FIG. 17 is a flowchart of an exemplary method 1600 for generating training data according to some embodiments of the present disclosure. Method 1600 may be performed by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual portion of method 1600 may be performed by any (or any combination of) one or more computing devices. Also, one or more portions of method 1600 may be implemented as an algorithm in the hardware components of the devices described herein (e.g., FIGS. 1-2, 4, 7, 18, etc.) for generating the training data described herein. FIG. 17 depicts elements performed in a particular order for purposes of illustration and discussion. Those skilled in the art will appreciate that, using the disclosure provided herein, elements of any of the methods described herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. FIG. 17 is described with reference to elements / terminology described in connection with other systems and figures for illustrative and illustrative purposes and is not intended to be limiting. One or more portions of method 1600 may additionally or alternatively be performed by other systems.

[0236] At 1605, method 1600 includes acquiring training scene data for the training environment at a first time step. For example, the training scene data may include real-world data or simulated data recorded by one or more sensors of the autonomous platform.

[0237] At step 1610, the method 1600 includes generating a plurality of actor labels and goal labels for the training environment, where the goal labels identify a plurality of goals for the training actors.

[0238] In step 1615, the method 1600 includes acquiring training scene data for the training environment over a time range that identifies the actual trajectory of the training actor.

[0239] In step 1620, the method 1600 includes calculating a Mean-Track Distance between the actual trajectory and the respective target path for each of the actor's multiple training targets.

[0240] In step 1625, the method 1600 includes updating a plurality of actor labels and target labels for the training environment.

[0241] 18 is a block diagram of an exemplary computing ecosystem 10 according to an exemplary embodiment of the present disclosure. The exemplary computing ecosystem 10 may include a first computing system 20 and a second computing system 40 communicatively coupled via one or more networks 60. In some embodiments, the first computing system 20 or the second computing system 40 may implement one or more of the systems, operations, or functions described herein for intent prediction (e.g., remote system 160, onboard computing system 180, autonomous system 200, etc.).

[0242] In some embodiments, the first computing system 20 may be included in an autonomous platform and may be utilized to perform the functions of the autonomous platform as described herein. For example, the first computing system 20 may be located onboard an autonomous vehicle and may implement an autonomous system for autonomously operating the autonomous vehicle. In some embodiments, the first computing system 20 may represent an entire onboard computing system or a portion thereof (e.g., a localization system 230, a perception system 240, a planning system 250, a control system 260, or a combination thereof). In other embodiments, the first computing system 20 may not be located onboard an autonomous platform. The first computing system 20 may include one or more separate physical computing devices 21.

[0243] The first computing system 20 (e.g., its computing device 21) may include one or more processors 22 and memory 23. The one or more processors 22 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 23 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

[0244] The memory 23 may store information that can be accessed by one or more processors 22. For example, the memory (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) may store data 24 that can be retrieved (e.g., received, accessed, written, manipulated, formed, generated, stored, pooled, downloaded, etc.). The data 24 may include, for example, sensor data, map data, data related to autonomous functions (e.g., data related to perception, planning, or control functions), simulation data, or any data or information described herein. In some embodiments, the first computing system 20 may retrieve data from one or more memory devices remote from the first computing system 20.

[0245] The memory 23 may store computer-readable instructions 25 that can be executed by one or more processors 22. The instructions 25 may be implemented in software written in any suitable programming language or in hardware. Additionally or alternatively, the instructions 25 may be executed in logically or virtually separated threads on the processors 22.

[0246] For example, memory 23 may store instructions 25 executable by one or more processors (e.g., one or more processors 22, one or more other processors) to perform any of the operations, functions, or methods / processes (or portions thereof) described herein (e.g., in computing device 21, first computing system 20, or other system having a processor that executes instructions).

[0247] In some embodiments, the first computing system 20 may store or include one or more models 26. In some embodiments, the model 26 may be or otherwise include one or more machine learning models. By way of example, the model 26 may be or otherwise include various machine learning models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models, including linear and nonlinear models. Examples of neural networks include forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system 20 may include one or more models for implementing subsystems of the autonomous system 200, including any of the localization system 230, perception system 240, planning system 250, or control system 260.

[0248] In some embodiments, first computing system 20 may obtain one or more models 26 that it communicates with second computing system 40 over network 60 using communications interface 27. For example, first computing system 20 may store models (e.g., one or more machine learning models) in memory 23. First computing system 20 can then use or otherwise implement (e.g., by processor 22) models 26. For example, first computing system 20 can implement models 26 to estimate the autonomous platform's position within an environment, recognize the autonomous platform's environment or objects therein, plan one or more future states of the autonomous platform for movement through the environment, and control the autonomous platform to interact with the environment.

[0249] The second computing system 40 may include one or more computing devices 41. The second computing system 40 may include one or more processors 42 and memory 43. The one or more processors 42 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 43 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

[0250] The memory 43 may store information accessible by the one or more processors 42. For example, the memory 43 (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) may store retrievable data 44. The data 44 may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data related to vehicle trips / services, or any data or information described herein. In some embodiments, the second computing system 40 may retrieve data from one or more memory devices remote from the second computing system 40.

[0251] The memory 43 may also store computer-readable instructions 45 that can be executed by one or more processors 42. The instructions 45 may be implemented in software written in any suitable programming language or in hardware. Additionally or alternatively, the instructions 45 may be executed in logically or virtually separate threads on the processors 42.

[0252] For example, memory 43 may store instructions 45 that are executable (e.g., by one or more processors 42, one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., by a computing device 41, such as computing device 21 or first computing system 20, second computing system 40, or other system having a processor for executing instructions). This may include, for example, functions of autonomous system 200 (e.g., localization, perception, planning, control, etc.) or other functions associated with an autonomous platform (e.g., remote assistance, mapping, vehicle management, trip / service assignment and matching, etc.).

[0253] In some embodiments, second computing system 40 may include one or more server computing devices. When second computing system 40 includes multiple server computing devices, the server computing devices may operate according to various computing architectures, including, for example, a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0254] In addition to or instead of the model 26 of the first computing system 20, the second computing system 40 may include one or more models 46. For example, the model 46 may be or otherwise include one or more machine learning models. By way of example, the model 26 may be or otherwise include various machine learning models such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models, including linear and nonlinear models. Examples of neural networks include forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system 40 may include one or more models of the autonomous system 200.

[0255] In some embodiments, second computing system 40 or first computing system 20 can train one or more machine learning models of model 26 or model 46 using one or more model trainers 47 and training data 48. Model trainer 47 can train either model 26 or model 46 using one or more training or learning algorithms. One example of a training technique is error backpropagation. In some embodiments, model trainer 47 can perform supervised training techniques using labeled training data. In other embodiments, model trainer 47 can perform unsupervised training techniques using unlabeled training data. In some embodiments, training data 48 can include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some embodiments, second computing system 40 can perform simulations to obtain training data 48 or to perform model trainer 47 to train or test model 26 or model 46. For example, model trainer 47 may train one or more components of a machine learning model for autonomous system 200 through unsupervised training techniques that use objective functions (e.g., costs, compensations, heuristics, constraints, etc.). In some embodiments, model trainer 47 may perform multiple generalization techniques to improve the generalization ability of the model being trained. Generalization techniques include weight decays, dropouts, or other techniques.

[0256] The first computing system 20 and the second computing system 40 may each include a communications interface 27, 49. The communications interfaces 27, 49 may be used to communicate with each other or with one or more other systems or devices, including systems or devices located remotely from the first computing system 20 or the second computing system 40. The communications interfaces 27, 49 may include any circuitry, components, software, etc. for communicating with one or more networks (e.g., network 60). In some embodiments, the communications interfaces 27, 49 may include, for example, one or more of a communications controller, a receiver, a transceiver, a transmitter, a port, a conductor, software, or hardware for data communication.

[0257] Network 60 may be any type of network or combination of networks that enables communication between devices. In some embodiments, the network may include one or more of a local area network, a wide area network, the Internet, a secure network, a cellular network, a mesh network, a peer-to-peer communication link, or some combination thereof, and may include any number of wired or wireless links. Communication over network 60 may be achieved, for example, via a network interface using any type of protocol, protection scheme, encoding, formatting, packaging, etc.

[0258] 18 illustrates an example computing ecosystem 10 that can be used to implement the present disclosure. Other systems can also be used. For example, in some embodiments, a first computing system 20 can include a model trainer 47 and training data 48. In such embodiments, the models 26, 46 can all be trained and used locally on the first computing system 20. As another example, in some embodiments, the computing system 20 may not be connected to other computing systems. Also, components illustrated or discussed as being included in one of the computing systems 20 or 40 can instead be included in the other one of the computing systems 20 or 40.

[0259] Computing tasks discussed herein as being performed on a computing device remote from the autonomous platform (e.g., an autonomous vehicle) may alternatively be performed on the autonomous platform (e.g., via a vehicle computing system of the autonomous vehicle), and vice versa. Such configurations may be implemented without departing from the scope of this disclosure. The use of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions among components. Computer-implemented operations may be performed on a single component or across several components. Computer-implemented tasks or operations may be performed sequentially or in parallel. Data and instructions may be stored in a single memory device or across multiple memory devices.

[0260] Aspects of the present disclosure have been described in terms of exemplary embodiments thereof. Those skilled in the art will be able to devise numerous other embodiments, modifications, or variations within the scope and spirit of the appended claims upon consideration of this disclosure. Any features recited in the following claims can be combined or rearranged in all possible ways. Accordingly, the scope of the present disclosure is exemplary rather than limiting, and the present disclosure does not exclude such variations, modifications, or additions to the present subject matter that would be obvious to those skilled in the art. It should be noted that terms are described herein using lists of exemplary elements joined by conjunctions such as "and," "or," "but," and the like. It should be understood that these conjunctions are provided for illustrative purposes only. For example, a list joined by a particular conjunction, such as "or," can mean "at least one" or "any combination" of the exemplary elements listed therein, and unless expressly stated otherwise, "or" should be understood as "and / or." Additionally, terms such as "based on" should be understood as "based at least in part on."

[0261] Those skilled in the art will understand, using the disclosure provided herein, that any elements of the claims, operations, or processes discussed herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Some claims are described with letter references to claim elements for illustrative purposes and not by way of limitation. The letter references do not imply a particular order of operations. For example, letter identifiers such as (a), (b), (c), ..., (i), (ii), (iii), ... may be used to describe operations. These identifiers are provided for the reader's convenience and do not indicate a particular order of steps or operations. Operations illustrated with list identifiers such as (a), (i), etc. may occur before, after, or in parallel with other operations illustrated with list identifiers such as (b), (ii), etc.

Claims

1. 1. A computer-implemented method comprising: (a) acquiring state data associated with a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in said environment; (b) determining a plurality of potential goals, including at least one potential goal for each of the plurality of actors, based on the state data and the map data, the potential goals including potential destination locations within the environment and a desired route to the potential destination location; (c) processing the state data, the map data, and the plurality of potential targets with a machine learning predictive model to determine (i) a predicted target for each of the plurality of actors, (ii) a predicted interaction between the each of the plurality of actors and other actors based on the predicted target, and (iii) a continuous trajectory for the each of the actors based on the predicted target; (d) initiating autonomous vehicle movement based on the predicted goal, the predicted interaction, or the continuous trajectory for the individual actor.

2. 2. The computer-implemented method of claim 1, wherein step (c) further comprises determining a probability of each potential target for the individual actor, and wherein the predicted target comprises the individual potential target having the highest probability.

3. the target path of the potential target includes a nominal path defined in the map data; the map data includes a plurality of map features associated with the potential landmark; The computer-implemented method of claim 1 or 2, wherein the plurality of map features comprises a plurality of waypoints for the nominal route defined in the map data.

4. The computer-implemented method of claim 1 , wherein the predicted interactions are further based on potential goals for other actors in the environment that intersect with the predicted goals for the individual actor.

5. The computer-implemented method of claim 4 , wherein the predicted interactions include probability distributions for one or more interaction types between the individual actor and the other actors.

6. 6. The computer-implemented method of claim 5, wherein the predicted probability of interaction between the individual actor and the other actor is based on one or more distances between the individual actor and a shared space associated with the predicted goal and the potential goal of the other actor.

7. 7. The computer-implemented method of claim 1, wherein the state data includes a plurality of actor states associated with the individual actor, the plurality of actor states indicating one or more positions, one or more velocities, or one or more headings for the individual actor at a current time and one or more past times.

8. (c) step generating a plurality of feature embeddings corresponding to the plurality of actors and the plurality of potential targets by processing the state data, the map data, and the plurality of potential targets with a first portion of the machine learning predictive model; and processing the plurality of feature embeddings to determine the predicted goals for the individual actors, the predicted interactions between the individual actors and the other actors based on the predicted goals, and the continuous trajectory of the individual actors based on the predicted goals.

9. 9. The computer-implemented method of claim 8, wherein the machine learning predictive model comprises a graph neural network including a plurality of nodes and a plurality of edges, wherein the plurality of nodes comprises: (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors; and (ii) a plurality of goal nodes corresponding individually to each potential goal of the plurality of potential goals, and the plurality of edges comprises: (iii) one or more actor-to-goal edges connecting each actor node with each goal node; and (iv) one or more goal-to-goal edges connecting at least two goal nodes of the plurality of goal nodes.

10. 10. The computer-implemented method of claim 9, wherein the plurality of feature embeddings comprises an actor feature embedding corresponding to an actor node, a goal feature embedding corresponding to a goal node, an actor-goal feature embedding corresponding to an actor-goal edge, and a goal-goal feature embedding corresponding to a goal-goal edge.

11. (c) step processing at least one of the actor feature embeddings or the actor-goal feature embeddings to determine the predicted goals for the individual actors; and processing the goal-goal feature embeddings to determine the predicted interactions between the individual actor and the other actors based on the predicted goals.

12. receiving vehicle movement data for the autonomous vehicle indicative of potential movements of the autonomous vehicle; generating conditional feature embeddings based on the potential movements of the autonomous vehicle; concatenating the conditional feature embedding with at least one of the plurality of feature embeddings; 10. The computer-implemented method of claim 8, further comprising: determining the predicted goals, the predicted interactions, and the continuous trajectories for the individual actors by processing the plurality of feature embeddings with a machine learning conditional predictive model.

13. (c) step performing one or more rounds of message passing to generate a plurality of updated feature embeddings; and determining the predicted goals, the predicted interactions, and the continuous trajectories for the individual actors based on the plurality of updated feature embeddings.

14. 1. A computing system comprising: one or more processors; 14. A computing system comprising one or more non-transitory computer-readable media storing executable instructions to cause the one or more processors to perform the operations of any one of claims 1 to 13.

15. 1. A control system for an autonomous vehicle, comprising: The control system includes: one or more processors; 14. A control system for an autonomous vehicle comprising one or more non-transitory computer-readable media storing instructions executable to cause the one or more processors to perform the operations of any one of claims 1 to 13.

Images