Target-based movement forecasting

Abstract

An exemplary aspect of the present disclosure relates to an exemplary computer-implemented method for predicting the intentions of actors in an environment. The exemplary method includes obtaining state data related to a plurality of actors in the environment and map data representing a plurality of lanes in the environment. The method includes determining a plurality of potential goals for each actor based on the state data and the map data. The method includes processing the state data, the map data, and the plurality of potential goals with a machine learning predictive model to determine (i) predicted goals for each actor of the plurality of actors, (ii) predicted interactions between each actor of the plurality of actors and other actors based on the predicted goals, and (iii) continuous trajectories for each actor based on the predicted goals.

Description

Technical Field

[0001] Claim of priority This application claims the benefit of U.S. Patent Application No. 18 / 147,316, filed on December 28, 2022, and U.S. Patent Application No. 18 / 471,960, filed on September 21, 2023, the entire contents of which are incorporated herein by reference.

Background Art

[0002] An autonomous platform can process data to recognize an environment in which the autonomous platform can travel. For example, an autonomous vehicle can use various sensors to recognize the surrounding environment and identify objects around the autonomous vehicle. The autonomous vehicle can identify an appropriate route through the recognized surrounding environment and travel along that route without human intervention or with minimal intervention.

Summary of the Invention

[0003] This disclosure describes improved intent prediction techniques that can be used in autonomous platforms for motion prediction and ultimately motion planning. The improved intent prediction techniques include machine learning models (e.g., graph neural networks) trained to generate a number of discrete intents and continuous motion outputs based on historical actor observations and map geometry for traffic scenes. These outputs include (i) goal probability (e.g., the probability that an actor on a road will follow a nominal path), (ii) interaction probability (e.g., the probability that an actor will yield or reverse yield to another actor), and (iii) a continuous goal-based trajectory of an actor. For example, an autonomous platform, such as a self-driving vehicle, can use these outputs to more accurately predict the future movements of actors in its environment while planning its own movements. In some cases, additional machine learning models (e.g., typified graph neural networks) can be used to adjust the outputs based on the autonomous platform's predicted movements. The technologies described here can improve the speed, efficiency, and accuracy of predicting the future movements of dynamic actors in traffic scenarios, and as a result, can improve the decision-making and response times of dynamic actors when planning the movements of autonomous platforms.

[0004] More specifically, this disclosure relates to a machine learning predictive model for predicting the future movement of actors in a traffic scene based on environmental state data and map data. Actors include both autonomous platforms and other dynamic objects in the traffic scene. State data may include current and historical observations such as location, velocity, and direction of movement for each actor in the scene. Map data can identify multiple lanes in the traffic scene and different lane features for each of the multiple lanes. Based on the actors' current state and map data, the model may determine a number of potential goals for individual actors in the traffic scene. Each goal may include a short-term goal (e.g., a potential goal within the next 5 seconds) and a target path to the target that each actor can access based on the actor's location in the traffic scene (e.g., multiple waypoints along one or more traffic lanes). The machine learning predictive model may process the state data and map data to determine the probability that an actor (i) will follow a target path and / or (ii) will interact with other actors in the traffic scene while following a target path. Furthermore, this model can determine (iii) the continuous trajectories of actors conditioned according to the target path.

[0005] A machine learning prediction model may include a graph neural network with multiple nodes and edges. Multiple nodes may include one or more target nodes of type target node and one or more actor nodes of type actor node. Multiple edges may include one or more actor-target edges of type actor-target edge and one or more target-target edges of type target-target edge. An actor-target edge may connect a particular actor to its potential goals, and a target-target edge may connect two goals associated in a "shared space" (e.g., the space where two corresponding goal paths intersect), which could lead to an interaction between two actors performing two goals, respectively.

[0006] Multiple nodes and edges may contain feature representations encoded using features different from state data and map data, based on their respective node and edge types. For example, an actor node may contain an actor feature representation encoded based on the current state and one or more past states of a particular actor. A target node may contain a target feature representation encoded based on the waypoint information of the corresponding target path. An actor-target edge may contain an actor-target feature representation encoded based on actor state information in the PathTangent Frame. Furthermore, a target-target edge may contain a target-target feature representation encoded based on actor state information for two actors that may have two goals related to the shared space. Message forwarding may be performed in several rounds to update the feature representations based on information from adjacent nodes and edges. The output may be determined from the updated feature representations.

[0007] Machine learning prediction models can be trained using labeled training data. Labeled training data may include auto-labeled ground-truth information that identifies (i) the ground-truth goal path followed by one or more actors in a traffic scene, (ii) whether or not an interaction occurred, and (iii) the type of interaction that occurred (e.g., which actor made a concession). Ground truth information can be algorithmically determined based on future sensor observations of the traffic scene. Machine learning prediction models can be trained as loss functions that minimize the distance between the output and the ground truth information.

[0008] Aspects of this disclosure may provide various technical improvements to robotics, machine learning, and intent prediction techniques. The improved intent prediction techniques described herein may provide technical solutions to the technical problem of robots predicting the movement of dynamic objects in real time. Thus, the systems and methods of this disclosure may provide an improved approach to planning autonomous driving movements.

[0009] The systems and methods described herein can provide substantial improvements to robotics, machine learning, and robot intent prediction techniques by accumulating and leveraging newly available information such as probabilistic goal-based paths for dynamic objects, probabilistic interaction relationships between dynamic objects, and continuous trajectories conditioned according to probabilistic goal-based paths and / or interaction relationships. Unlike conventional intent prediction techniques, the systems and methods described herein output several discrete intent and continuous motion outputs that can be used to train machine learning models to accurately predict the movement of dynamic objects in an environment. In this way, the systems and methods described herein can improve the speed, efficiency, and accuracy of future object position prediction. This ultimately enhances the capabilities of robotics and machine learning techniques by enabling autonomous platforms to predict the future movement of objects in an environment. Finally, the techniques disclosed herein improve motion planning techniques and road driving performance for robots or autonomous vehicle technologies of various configurations by implementing more accurate, robust, and efficient object intent prediction techniques.

[0010] For example, in one embodiment, the Disclosure provides an exemplary Computer-Implemented Method. The exemplary Method includes the step of obtaining state data relating to a plurality of actors in an environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The Method includes the step of determining a plurality of potential targets, including at least one potential target for each of the plurality of actors, based on the state data and the map data. The potential targets include the location of a potential destination in the environment and a target path to the location of the potential destination. The Method includes the step of determining (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the individual actors of the plurality of actors and other actors based on the predicted targets, and (iii) sequential trajectories for the individual actors based on the predicted targets, by processing the state data, the map data and the plurality of potential targets using a machine learning predictive model. The Method includes the step of initiating the movement of the autonomous vehicle based on the predicted targets, the predicted interactions, or the sequential trajectories for the individual actors.

[0011] In some embodiments of the exemplary method described above, the method includes the step of determining the probability of each potential target for each individual actor. The predicted targets include the individual potential targets having the highest probability.

[0012] In some embodiments of the exemplary method described above, the state data is associated with at least one of the autonomous vehicle's localization system or inertial measurement device.

[0013] In some embodiments of the exemplary method described above, the target path of the potential target includes a nominal path defined in the map data.

[0014] In some embodiments of the exemplary method described above, the map data includes a plurality of map features related to the potential target. The plurality of map features also includes a plurality of waypoints for the nominal route defined in the map data.

[0015] In some embodiments of the exemplary method described above, the predicted interactions are further based on potential targets for other actors that intersect with the predicted targets for the individual actors in the environment.

[0016] In some embodiments of the exemplary method described above, the predicted interaction includes a probability distribution for one or more interaction types between the individual actors and the other actors.

[0017] In some embodiments of the exemplary method described above, the probability of the predicted interaction between the individual actors and the other actors is based on one or more distances between the individual actors and a shared space relating to the predicted target and the other actors' potential targets.

[0018] In some embodiments of the exemplary method described above, the state data includes a plurality of actor states associated with the individual actor. The plurality of actor states indicate one or more positions, one or more velocities, or one or more headings for the individual actor at the present time and one or more past time points.

[0019] In some embodiments of the exemplary method, the step of processing the state data, the map data, and the plurality of potential targets using a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the plurality of actors and other actors based on the predicted targets, and (iii) a continuous trajectory for each of the actors based on the predicted targets includes the steps of: generating a plurality of feature embeddings corresponding to the plurality of actors and the plurality of potential targets by processing the state data, the map data, and the plurality of potential targets in a first part of the machine learning predictive model; and processing the plurality of feature embeddings to determine the predicted targets for each of the actors, the predicted interactions between the plurality of actors and other actors based on the predicted targets, and the continuous trajectory for each of the actors based on the predicted targets.

[0020] In some embodiments of the exemplary method, the step of processing the state data, the map data, and the plurality of potential targets using a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the plurality of actors and other actors based on the predicted targets, and (iii) sequential trajectories for the individual actors based on the predicted targets, includes the steps of performing one or more message forwarding rounds to generate a plurality of updated feature embeddings, and determining the predicted targets, the predicted interactions, and the sequential trajectories for the individual actors based on the plurality of updated feature embeddings.

[0021] In some embodiments of the exemplary method described above, the machine learning prediction model includes a graph neural network comprising a plurality of nodes and a plurality of edges, wherein the plurality of nodes includes (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors, and (ii) a plurality of target nodes corresponding individually to each of the plurality of potential targets, and the plurality of edges includes (iii) one or more actor-target edges that individually connect individual actor nodes to individual target nodes, and (iv) one or more target-target edges that individually connect at least two of the plurality of target nodes.

[0022] In some embodiments of the exemplary method described above, the plurality of feature embeddings include actor feature embeddings corresponding to actor nodes, target feature embeddings corresponding to target nodes, actor-target feature embeddings corresponding to actor-target edges, and target-target feature embeddings corresponding to target-target edges.

[0023] In some embodiments of the exemplary method, the step of determining (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between each of the plurality of actors and other actors based on the predicted targets, and (iii) sequential trajectories for each of the actors based on the predicted targets, by processing the state data, the map data and the plurality of potential targets using a machine learning predictive model, includes the steps of processing at least one of the actor feature embeddings or the actor-target feature embeddings to determine the predicted targets for each of the actors, and processing the target-target feature embeddings to determine the predicted interactions between each of the actors and other actors based on the predicted targets.

[0024] In some embodiments of the exemplary method, the method includes receiving vehicle movement data for the autonomous vehicle. The vehicle movement data indicates potential movement of the autonomous vehicle.

[0025] In some embodiments of the exemplary method, the method includes generating a conditional feature embedding based on the potential movement of the autonomous vehicle.

[0026] In some embodiments of the exemplary method, the method further includes concatenating the conditional feature embedding with at least one of the plurality of feature embeddings.

[0027] In some embodiments of the exemplary method, the method further includes determining the predicted goal, the predicted interaction, and the continuous trajectory for the individual actor by processing the plurality of feature embeddings using a machine learning conditional prediction model.

[0028] For example, in one aspect, the present disclosure provides an exemplary computing system. The computing system includes one or more processors and one or more non-transitory computer-readable media storing executable instructions for the one or more processors to execute operations. The operations include obtaining state data related to a plurality of actors in the environment of an autonomous vehicle and map data representing a plurality of lanes of the environment. The operations include determining, for each of the plurality of actors, a plurality of potential goals including at least one potential goal based on the state data and the map data. The potential goals include the position of a potential destination in the environment and a target route to the position of the potential destination. The operations include processing the state data, the map data, and the plurality of potential goals using a machine learning prediction model to determine (i) a predicted goal for each individual actor of the plurality of actors, (ii) a predicted interaction between each individual actor of the plurality of actors and other actors based on the predicted goal, and (iii) a continuous trajectory for each individual actor based on the predicted goal. The operations include initiating movement of the autonomous vehicle based on the predicted goal, the predicted interaction, or the continuous trajectory for the individual actor.

[0029] In some embodiments of the exemplary computing system, processing the state data, the map data, and the plurality of potential goals using a machine learning prediction model to determine (i) a predicted goal for each individual actor of the plurality of actors, (ii) a predicted interaction between each individual actor of the plurality of actors and other actors based on the predicted goal, and (iii) a continuous trajectory for each individual actor based on the predicted goal further includes determining a probability for each potential goal for the individual actor. The predicted goal includes the individual potential goal having the highest probability.

[0030] In some embodiments of the exemplary computing system described above, the target path of the potential target includes a nominal path defined in the map data.

[0031] In some embodiments of the exemplary computing system, the map data includes a plurality of map features related to the potential target. The plurality of map features includes a plurality of waypoints for the nominal route defined in the map data.

[0032] For example, in one embodiment, the Disclosure provides an exemplary control system for an autonomous vehicle. The control system includes one or more processors and one or more non-temporary computer-readable media that store executable instructions enabling the one or more processors to perform operations. The operations include obtaining state data relating to a plurality of actors in the environment of the autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include determining a plurality of potential targets, each of the plurality of actors, based on the state data and the map data, including at least one potential target for each of the plurality of actors. The potential targets include the location of a potential destination in the environment and the target path to the location of the potential destination. The operations include processing the state data, the map data and the plurality of potential targets using a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the individual actors of the plurality of actors and other actors based on the predicted targets, and (iii) sequential trajectories for the individual actors based on the predicted targets. The aforementioned action includes an action to initiate the movement of the autonomous vehicle based on the predicted target, the predicted interaction, or the continuous trajectory relative to the individual actors.

[0033] In some embodiments of the exemplary control system, the operation of processing the state data, the map data, and the plurality of potential targets using a machine learning predictive model to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between the plurality of actors and other actors based on the predicted targets, and (iii) sequential trajectories for the individual actors based on the predicted targets, further includes the operation of determining the probability of each potential target for the individual actors. The predicted targets include the individual potential targets having the highest probability.

[0034] For example, in one embodiment, the Disclosure provides an exemplary computer implementation method. The computer implementation method includes the step of (a) obtaining state data relating to a plurality of actors in the environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The computer implementation method includes the step of (b) determining a plurality of potential targets based on the state data and the map data, including at least one potential target for each of the plurality of actors - the potential targets include the location of a potential destination in the environment and the target path to the location of the potential destination. The computer implementation method includes the step of (c) generating a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential targets by processing the state data, the map data and the plurality of potential targets using a machine learning predictive model. The computer implementation method includes the step of (d) processing the plurality of feature embeddings to determine predicted targets for individual actors of the plurality of actors and a continuum for the individual actors based on the predicted targets. The computer implementation method includes the step of (e) initiating the movement of the autonomous vehicle based on the predicted targets and the continuum for the individual actors.

[0035] In some embodiments of the exemplary computer implementation method, the machine learning prediction model includes the graph neural network having the individual nodes corresponding to the plurality of actors and the plurality of potential goals.

[0036] In some embodiments of the exemplary computer implementation method described above, the graph neural network further includes one or more actor-target edges that individually connect individual actor nodes to individual target nodes.

[0037] In some embodiments of the exemplary computer implementation method, the graph neural network further includes one or more goal-to-goal edges that individually connect at least two goal nodes of the individual nodes corresponding to the plurality of potential goals.

[0038] In some embodiments of the exemplary computer implementation method described above, the plurality of feature embeddings include actor feature embeddings corresponding to actor nodes, target feature embeddings corresponding to target nodes, actor-target feature embeddings corresponding to actor-target edges, and target-target feature embeddings corresponding to target-target edges.

[0039] In some embodiments of the exemplary computer implementation method, the method further includes the step of determining the probability of each potential target of the individual actors, wherein the predicted targets include the individual potential targets having the highest probability.

[0040] In some embodiments of the exemplary computer implementation method described above, the target path of the at least one potential target includes a nominal path defined in the map data.

[0041] In some embodiments of the exemplary computer implementation method, the at least one potential target for each actor comprises a plurality of potential targets for the actor, and the predicted target is a probability distribution for the plurality of potential targets.

[0042] In some embodiments of the exemplary computer implementation method, step (d) includes processing the plurality of feature embeddings to determine the predicted interactions between the individual actors and other actors of the plurality of actors based on the predicted goals.

[0043] In some embodiments of the exemplary computer implementation method, the predicted interactions are further based on potential targets for other actors that intersect with the predicted targets for the individual actors in the environment.

[0044] In some embodiments of the exemplary computer implementation method described above, the predicted interaction includes a probability distribution for one or more interaction types between the individual actors and the other actors.

[0045] In some embodiments of the exemplary computer implementation method, the probability of the predicted interaction between the individual actors and the other actors is based on one or more distances between the individual actors and the shared space relating to the predicted target and the potential target for the other actors.

[0046] In some embodiments of the exemplary computer implementation method described above, the method includes the step of performing one or more message transfer rounds to generate a plurality of updated feature embeddings. The method further includes the step of determining the predicted target and the sequential trajectory for the individual actors based on the plurality of updated feature embeddings.

[0047] For example, in one embodiment, the Disclosure provides an exemplary computing system. The computing system includes one or more processors and one or more non-temporary computer-readable media for storing executable instructions that enable the one or more processors to perform operations. The operations include (a) acquiring state data relating to a plurality of actors in the environment of an autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include (b) determining a plurality of potential targets based on the state data and the map data, including at least one potential target for each of the plurality of actors—the potential targets including the location of a potential destination in the environment and the target path to the location of the potential destination. The operations include (c) generating a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential targets by processing the state data, the map data and the plurality of potential targets using a machine learning predictive model. The operations include (d) processing the plurality of feature embeddings to determine predicted targets for individual actors of the plurality of actors and a continuum for individual actors based on the predicted targets. The operation includes (e) an operation in which the autonomous vehicle starts moving based on the predicted target and the continuous trajectory of each of the actors.

[0048] In some embodiments of the exemplary computing system, the machine learning prediction model includes the graph neural network having individual nodes corresponding to the plurality of actors and the plurality of potential goals.

[0049] In some embodiments of the exemplary computing system, the graph neural network includes one or more actor-target edges that individually connect individual actor nodes to individual target nodes.

[0050] In some embodiments of the exemplary computing system, the graph neural network includes one or more goal-to-goal edges that individually connect at least two goal nodes of the individual nodes corresponding to the plurality of potential goals.

[0051] In some embodiments of the exemplary computing system described above, the operation includes performing one or more message forwarding rounds to generate a plurality of updated feature embeddings.

[0052] In some embodiments of the exemplary computing system, the operation includes determining the predicted target and the sequential trajectory for each actor based on the plurality of updated feature embeddings.

[0053] In some embodiments of the exemplary computing system, (d) includes the operation of processing the plurality of feature embeddings to determine (i) predicted targets for each of the plurality of actors, (ii) predicted interactions between each of the plurality of actors and other actors based on the predicted targets, and (iii) a continuous trajectory for each of the individual actors based on the predicted targets and the predicted interactions.

[0054] For example, in one embodiment, the Disclosure provides an exemplary control system for an autonomous vehicle. The control system includes one or more processors and one or more non-temporary computer-readable media for storing executable instructions that enable the one or more processors to perform operations. The operations include (a) obtaining state data relating to a plurality of actors in the environment of the autonomous vehicle and map data representing a plurality of lanes in the environment. The operations include (b) determining a plurality of potential targets based on the state data and the map data, including at least one potential target for each of the plurality of actors - the potential targets including the location of a potential destination in the environment and the target path to the location of the potential destination. The operations include (c) generating a plurality of feature embeddings for individual nodes of a graph neural network corresponding to the plurality of actors and the plurality of potential targets by processing the state data, the map data and the plurality of potential targets using a machine learning predictive model. The operations include (d) processing the plurality of feature embeddings to determine predicted targets for individual actors of the plurality of actors and a continuum for individual actors based on the predicted targets. The operation includes (e) an operation in which the autonomous vehicle starts moving based on the predicted target and the continuous trajectory of each of the actors.

[0055] Other exemplary embodiments of this disclosure relate to other systems, methods, vehicles, apparatus, tangible non-temporary computer-readable media and devices for generating trajectories, learning models, and performing other functions described herein. These and other features, aspects and advantages of various embodiments will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and constituting part thereof are used to illustrate embodiments of this disclosure and to illustrate relevant principles together with the description. [Brief explanation of the drawing]

[0056] A detailed description of embodiments presented to those skilled in the art is given herein with reference to the accompanying drawings.

[0057] [Figure 1] This is a block diagram of an operating scenario according to some embodiments of the present disclosure.

[0058] [Figure 2] This is a block diagram of a system according to some embodiments of this disclosure.

[0059] [Figure 3a] This figure shows the operating environment according to some embodiments of the present disclosure.

[0060] [Figure 3b] This figure shows a map of the operating environment according to some embodiments of this disclosure.

[0061] [Figure 3c] This figure shows the operating environment according to some embodiments of the present disclosure.

[0062] [Figure 3d] This figure shows a map of the operating environment according to some embodiments of this disclosure.

[0063] [Figure 4] This is a block diagram of a system according to some embodiments of the present disclosure for initiating the operation of an autonomous platform.

[0064] [Figure 5] This is an exemplary environment that can be transformed for input to a prediction system by some embodiments of this disclosure.

[0065] [Figure 6] This is a graph representation of a graph neural network (GNN) generated using a machine learning prediction model according to some embodiments of this disclosure.

[0066] [Figure 7] This is a block diagram of an exemplary system architecture of a prediction system according to some embodiments of the present disclosure.

[0067] [Figure 8] This is an illustrative diagram of the data flow within a machine learning predictive model according to some embodiments of the present disclosure.

[0068] [Figure 9] This is an exemplary environment that is transformed for input to a prediction system by some embodiments of this disclosure.

[0069] [Figure 10] This diagram illustrates the data flow related to a continuous trajectory in a machine learning predictive model, according to some embodiments of this disclosure.

[0070] [Figure 11] This is an illustrative diagram illustrating the generation of labeled training data for training a machine learning predictive model according to some embodiments of the present disclosure.

[0071] [Figure 12] This is an illustrative diagram illustrating the generation of labeled training data for training a machine learning predictive model according to some embodiments of the present disclosure.

[0072] [Figure 13] This is a flowchart illustrating an exemplary method according to some embodiments of the present disclosure for initiating the movement of an autonomous vehicle.

[0073] [Figure 14] This is an illustrative flowchart of a data processing method using a machine learning prediction model according to some embodiments of this disclosure.

[0074] [Figure 15] This is an illustrative flowchart of a data processing method using a machine learning prediction model according to some embodiments of this disclosure.

[0075] [Figure 16] This is a flowchart illustrating an exemplary method according to some embodiments of the present disclosure for training a machine learning predictive model.

[0076] [Figure 17] This is a flowchart illustrating an exemplary method according to some embodiments of the present disclosure for generating training data.

[0077] [Figure 18] This is a block diagram of a computing system according to some embodiments of the present disclosure for predicting the intent of one or more objects in an environment. [Modes for carrying out the invention]

[0078] In the following, the technologies of this disclosure will be described in the context of autonomous vehicles only for illustrative purposes. As described herein, the technologies of this disclosure are not limited to autonomous vehicles but can be implemented for or within autonomous robots, other autonomous platforms, and other computing systems.

[0079] Exemplary embodiments of the present disclosure will be described in more detail with reference to Figures 1 to 17. Figure 1 is a block diagram of an exemplary operating scenario according to some embodiments of the present disclosure. In the exemplary operating scenario, the environment 100 includes an autonomous platform 110 and a plurality of objects, including a first actor 120, a second actor 130, and a third actor 140. In the exemplary operating scenario, the autonomous platform 110 can move through the environment 100 and interact with objects located within the environment 100 (e.g., the first actor 120, the second actor 130, the third actor 140, etc.). The autonomous platform 110 may be selectively configured to communicate with a remote system 160 via a network 170.

[0080] Environment 100 is or may include an indoor environment (e.g., an environment within one or more facilities) or an outdoor environment. An indoor environment may be an environment enclosed by a structure such as a building (e.g., a service warehouse, maintenance area, manufacturing facility). An outdoor environment may be one or more areas of the outside world such as one or more rural areas (e.g., having one or more rural travel routes), one or more urban areas (e.g., having one or more urban travel routes, highways, etc.), one or more suburban areas (e.g., having one or more suburban travel routes), or other outdoor environments.

[0081] The autonomous platform 110 can be any type of platform configured to operate within the environment 100. For example, the autonomous platform 110 can be a vehicle configured to autonomously perceive and operate within the environment 100. The vehicle can be, for example, a ground-based autonomous vehicle such as an autonomous vehicle, truck, or van. The autonomous platform 110 can be an autonomous vehicle that can control, be coupled to, or associate with tools, attachments and / or accessories for transporting people or cargo. The autonomous platform can include, for example, an autonomous tractor that is selectively coupled to a cargo trailer. Additionally or alternatively, the autonomous platform 110 can be any other type of vehicle, such as one or more aircraft, water vehicles, space-based vehicles, or other ground-based vehicles.

[0082] The autonomous platform 110 may be configured to communicate with a remote system 160. For example, the remote system 160 may communicate with the autonomous platform 110 for assistance (e.g., mobility assistance, situational assistance, etc.), control (e.g., vehicle management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, the remote system 160 may provide data instructing the autonomous platform 110 on tasks to be performed. For example, as further described herein, the remote system 160 may provide data instructing the autonomous platform 110 that it must perform trips / services such as user transport trips / services, delivery trips / services (e.g., cargo, goods, articles), etc.

[0083] The autonomous platform 110 may communicate with the remote system 160 using the network 170. The network 170 can facilitate the transmission of signals (e.g., electronic signals) or data (e.g., data from a computing device) and may include various wired (e.g., twisted-pair cables) or wireless communication mechanisms (e.g., cellular, radio, satellite, microwave, radio frequency, etc.) or any combination of any desired network topology (or topology). For example, the network 170 may include a local area network (e.g., an intranet), a wide area network (e.g., the Internet), a wireless LAN network (e.g., via WiFi, etc.), a cellular network, a SATCOM network, a VHF network, an HF network, a WiMAX-based network, or any other suitable communication network (or a combination thereof) for transmitting data to or from the autonomous platform 110.

[0084] For example, as shown in Figure 1, the environment 100 may include one or more objects. The objects may be objects that do not move or are not expected to move ("static objects"), or objects that move or are expected to move ("dynamic objects" or "actors"). In some embodiments, the environment 100 may include any number of actors, such as one or more pedestrians, animals, vehicles, etc. Actors can move within the environment along one or more actor trajectories. For example, a first actor 120 can move along any one of the first actor trajectories 122A to C, a second actor 130 can move along any one of the second actor trajectories 132, and a third actor 140 can move along any one of the third actor trajectories 142.

[0085] As further described herein, the autonomous platform 110 may utilize its own autonomous system to detect and plan the movements of actors (and their movements) as it travels through the environment 100 along one or more platform trajectories 112A-C. The autonomous platform 110 may include an onboard computing system 180. The onboard computing system 180 may include one or more processors and one or more memory devices. The one or more memory devices may store instructions executable by one or more processors, enabling one or more processors to perform operations or functions related to the autonomous platform 110, including the implementation of the autonomous driving system of the autonomous platform.

[0086] Figure 2 is a block diagram of an exemplary autonomous system 200 for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system 200 can be implemented by the computing system of the autonomous platform (e.g., the onboard computing system 180 of the autonomous platform 110). The autonomous system 200 can operate to acquire input from sensors 202 or other input devices. In some embodiments, the autonomous system 200 may further acquire platform data 208 (e.g., map data 210) from local or remote storage. The autonomous system 200 may generate control outputs for controlling the autonomous platform (e.g., via a platform control device 212, etc.) based on the sensor data 204, map data 210, or other data. The autonomous system 200 may include different subsystems for performing various autonomous operations. The subsystems may include a position estimation system 230, a perception system 240, a planning system 250, and a control system 260. The position estimation system 230 can determine the position of the autonomous platform in the environment, the recognition system 240 can detect, classify, and track objects and actors in the environment, the planning system 250 can determine the trajectory of the autonomous platform, and the control system 260 can translate the trajectory into vehicle control actions for controlling the autonomous platform. The autonomous system 200 can be implemented by one or more onboard computing systems. Subsystems may include one or more processors and one or more memory devices. One or more memory devices can store instructions executable by one or more processors so that one or more processors can perform operations or functions related to the subsystem. The computing resources of the autonomous system 200 may be shared among its subsystems, or subsystems may have their own dedicated sets of computing resources.

[0087] In some embodiments, the autonomous system 200 can be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomous system 200 can perceive and understand the vehicle's surrounding environment and perform various processing techniques on inputs (e.g., sensor data 204, map data 210) to generate an appropriate set of control outputs for implementing a vehicle motion plan (e.g., including one or more trajectories) to navigate the vehicle's surrounding environment (e.g., environment 100 in Figure 1). In some embodiments, an autonomous vehicle implementing the autonomous system 200 can drive, operate, and operate with minimal or no interaction with a human operator (e.g., driver, pilot, etc.).

[0088] In some embodiments, the autonomous platform may be configured to operate in multiple operating modes. For example, the autonomous platform may be configured to operate in a fully autonomous (e.g., autonomous driving) operating mode, where the autonomous platform can be controlled without user input (e.g., without input from a human operator present in or remotely from the autonomous vehicle). The autonomous platform may operate in a semi-autonomous driving operating mode, where the autonomous platform can operate with some input from a human operator present in the autonomous platform (or a human operator remotely from the autonomous platform). In some embodiments, the autonomous platform may enter a manual operating mode, where the autonomous platform can be fully controlled by a human operator (e.g., a human driver), and autonomous driving (e.g., autonomous driving) can be prohibited or deactivated (e.g., temporarily, permanently, etc.). The autonomous platform may be configured to operate in other modes, such as a parking or power-saving mode (e.g., used during operations such as trip / service standby, charging, etc.). In some embodiments, the autonomous platform may implement vehicle operation assistance technologies (e.g., collision mitigation systems, power-assisted steering, etc.) to assist the human operator of the autonomous platform, for example (e.g., while in manual mode, etc.).

[0089] The autonomous system 200 may be located onboard the autonomous platform (e.g., on or inside the autonomous platform) and may be configured to operate the autonomous platform in various environments. The environment may be a real environment or a simulated environment. In some embodiments, one or more simulation computing devices may simulate one or more of the following to simulate the operation of the autonomous system 200: sensors 202, sensor data 204, communication interfaces 206, platform data 208, or platform control devices 212.

[0090] In some embodiments, the autonomous system 200 may communicate with one or more networks or other systems via a communication interface 206. The communication interface 206 may include, for example, a transmitter, receiver, port, controller, antenna, or any other suitable components that can facilitate communication, and may include any components suitable for interfacing with one or more networks (e.g., network 170 in Figure 1). In some embodiments, the communication interface 206 may include multiple components (e.g., antenna, transmitter, or receiver) that enable the implementation and utilization of various communication technologies (e.g., multiple input, multiple output (MIMO) technology).

[0091] In some embodiments, the autonomous system 200 may communicate with one or more remote computing devices from an autonomous platform (e.g., remote system 160) via one or more networks (e.g., network 170) using a communication interface 206. For example, in some embodiments, one or more inputs, data, or functions of the autonomous system 200 may be complemented or replaced by a remote system communicating via the communication interface 206. For example, in some embodiments, map data 210 may be downloaded to a remote system via a network using the communication interface 206. In some embodiments, one or more of the position estimation system 230, recognition system 240, planning system 250, or control system 260 may be updated, influenced, guided, communicated, etc., by a remote system for support, maintenance, situational redefinition, management, etc.

[0092] Sensor 202 may be onboard the autonomous platform. In some embodiments, sensor 202 may include one or more types of sensors. For example, one or more sensors may include image capture devices (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally or alternatively, sensor 202 may include one or more depth capture devices. For example, sensor 202 may include one or more light detection and distance measuring (LIDAR) sensors or radio detection and distance measuring (RADAR) sensors. Sensor 202 may be configured to generate point data describing at least a portion of a 360-degree view of the surrounding environment. The point data may be point cloud data (e.g., 3D LIDAR point cloud data, RADAR point cloud data). In some embodiments, one or more of the sensors 202 for capturing depth information may be fixed to a rotating device to rotate the sensor 202 around an axis. While rotating around the axis, sensor 202 may capture data in spaced sector packets describing other portions of a 360-degree view of the surrounding environment of the autonomous platform. In some embodiments, one or more of the sensors 202 for capturing depth information may be solid-state sensors.

[0093] Sensor 202 may be configured to capture sensor data 204 representing or related to at least a portion of the environment of the autonomous platform. Sensor data 204 may include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system 200 may receive input from additional types of sensors, such as inertial measuring units (IMUs), altimeters, inclinometers, odometers, positioning or location devices (e.g., GPS, compasses), wheel encoders, or other types of sensors. In some embodiments, the autonomous system 200 may receive sensor data 204 related to specific components or systems of the autonomous platform. This sensor data 204 may indicate, for example, wheel speed, component temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system 200 may receive sensor data 204 related to ambient conditions, such as environmental or weather conditions. In some embodiments, sensor data 204 may include multi-modal sensor data. Multimodal sensor data can be acquired by at least two types of sensors (e.g., sensors in sensor 202) and can represent static objects or actors in the environment of the autonomous platform. Multimodal sensor data may include at least two types of sensor data (e.g., camera and LIDAR data). In some embodiments, the autonomous platform can leverage sensor data 204 for sensors located remotely from the autonomous platform (e.g., off-board). This may include, for example, sensor data 204 captured by another autonomous platform.

[0094] The autonomous system 200 may acquire map data 210 related to the environment in which the autonomous platform is located, is located, or will be located in the future. The map data 210 may provide information about the environment or geographical area. For example, map data 210 may provide information on the identification and location of various travel routes (e.g., roads), travel route segments (e.g., road segments), buildings or other items or objects (e.g., streetlights, crosswalks, curbs), the location and orientation of boundaries or boundary markers (e.g., location and orientation of traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.), traffic control data (e.g., location and indication of signs, traffic lights, other traffic control devices), obstacle information (e.g., temporary or permanent blockages), event data (e.g., road closures / changes in traffic rules due to parades, concerts, sporting events, etc.), nominal vehicle route data (e.g., representing an ideal vehicle route, such as one aligned with the center of a particular lane), or any other map data that provides information to help the autonomous platform understand the surrounding environment and its relationships. In some embodiments, map data 210 may include high-resolution map information. Furthermore, or alternatively, map data 210 may include scarce map data (e.g., lane graphs, etc.). In some embodiments, the sensor data 204 can be fused with the map data 210 or used to update the map data 210 in real time.

[0095] The autonomous system 200 may include a position estimation system 230 that can provide the autonomous platform with an understanding of its position and orientation in the environment. In some examples, the position estimation system 230 can support one or more other subsystems of the autonomous system 200 by, for example, providing an integrated local reference frame for performing recognition, planning, or control operations.

[0096] In some embodiments, the position estimation system 230 may determine the current position of the autonomous platform. The current position may include an absolute position (Global Position) (based on, for example, georeference anchors) or a relative position (based on, for example, objects in the environment). The position estimation system 230 may generally include or interface with any device or circuit for analyzing the position or change in position of the autonomous platform (e.g., an autonomous ground-based vehicle). For example, the position estimation system 230 may determine the position by using one or more of the following: inertial sensors (e.g., inertial measuring devices), satellite position estimation systems, radio receivers, networking devices (e.g., based on IP addresses), network access points or other network components (e.g., cellular towers, WiFi access points), or other suitable techniques. The position of the autonomous platform may be used by various subsystems of the autonomous system 200 or provided to a remote computing system (e.g., using a communication interface 206).

[0097] In some embodiments, the position estimation system 230 can register the relative positions of elements in the surrounding environment of the autonomous platform along with positions recorded in the map data 210. For example, the position estimation system 230 can process sensor data 204 (e.g., LIDAR data, RADAR data, camera data, etc.) for alignment or other registration on a map of the surrounding environment (e.g., from the map data 210) to determine the position of the autonomous platform in that environment. Thus, in some embodiments, the autonomous platform can identify its position in the surrounding environment (e.g., across six axes) based on a lookup of the map data 210. In some embodiments, given an initial position, the position estimation system 230 can update the position of the autonomous platform by incremental realignment based on deviations recorded or estimated from the initial position. In some embodiments, the position can be registered directly in the map data 210.

[0098] In some embodiments, the map data 210 may include a large amount of data subdivided into geographic tiles so that a desired area of ​​the map stored in the map data 210 can be reconstructed from one or more tiles. For example, multiple tiles selected from the map data 210 may be stitched together by the autonomous system 200 based on locations obtained by the location estimation system 230 (e.g., multiple tiles selected near a location).

[0099] In some embodiments, the position estimation system 230 may determine the position (e.g., relative or absolute position) of one or more attachments or accessories relative to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and the position estimation system 230 may provide the position of one or more points on the cargo platform. For example, the cargo platform may include a trailer or other device towed, otherwise attached to, or operated by the autonomous platform, and the position estimation system 230 may provide data describing the position (e.g., absolute, relative position, etc.) of the autonomous platform as well as the cargo platform. This information may be acquired by other autonomous driving systems to assist the movement of the autonomous platform.

[0100] The autonomous system 200 may include a recognition system 240 that enables the autonomous platform to detect, classify, and track objects and actors in the environment. Environmental features or objects recognized in the environment may be those that are within the field of view of sensor 202 or that are expected to be hidden from sensor 202. This may include objects that do not move or are not expected to move (static objects), or objects that move or are expected to move (dynamic objects / actors).

[0101] The recognition system 240 may determine one or more states (e.g., current or past states) of one or more objects in the surrounding environment of the autonomous platform. For example, a state may describe the object's current or past position (also called position) (e.g., for a given time, period, etc.), current or past speed / velocity, current or past acceleration, current or past direction of travel, current or past orientation, size / footprint (e.g., represented by boundary shape, object highlighting, etc.), classification (e.g., pedestrian class vs. vehicle class vs. bicycle class), associated uncertainty, or estimates of other state information. In some embodiments, the recognition system 240 may determine states using one or more algorithms or machine learning models configured to identify / classify objects based on input from sensors 202. The recognition system may use different forms of sensor data 204 to generate representations of the environment that are processed by one or more algorithms or machine learning models. In some embodiments, the state of one or more identified or unidentified objects may be maintained and updated over time as the autonomous platform continues to recognize the objects and interact with them (e.g., launch, make concessions, etc., with or around the objects). In this way, the recognition system 240 may provide an understanding of the current state of a known environment (e.g., a state including objects in the environment) from a record of the environment's previous state (e.g., a state including a movement history of objects in the environment). This information may be useful when the autonomous platform plans its movements through the environment.

[0102] The autonomous system 200 may include a planning system 250 which may be configured to determine how the autonomous platform interacts with and moves within the environment. The planning system 250 may determine one or more motion plans for the autonomous platform. A motion plan may include one or more trajectories (e.g., motion trajectories) that indicate the path the autonomous platform should follow. A trajectory may be of a specific length or time range. The length or time range may be defined by the calculation planning range of the planning system 250. A motion trajectory may be defined by one or more waypoints (including associated coordinates). A waypoint may be the future location of the autonomous platform. Motion plans may be continuously generated, updated, and reviewed by the planning system 250.

[0103] The motion planning system 250 can determine a strategy for the autonomous platform. The strategy may be a series of individual decisions made by the autonomous platform (e.g., concessions to actors, counter-concessions to actors, merging, lane changes). The strategy may be selected from several potential strategies. The selected strategy may be the lowest-cost strategy determined by one or more cost functions. These cost functions may, for example, evaluate the likelihood of collisions with other actors or objects.

[0104] The planning system 250 may determine a preferred trajectory for executing a strategy. For example, the planning system 250 may obtain one or more trajectories for executing one or more strategies. The planning system 250 may evaluate and rank the trajectories or strategies (e.g., using scores, costs, compensations, constraints, etc.). For example, the planning system 250 may provide an evaluation of candidate trajectories or strategies for the autonomous platform using predictive outputs representing interactions between the autonomous platform's trajectory and one or more objects (e.g., proximity, intersections, etc.). In some embodiments, the planning system 250 may leverage static costs to evaluate the trajectory of the autonomous platform (e.g., "avoid lane boundaries," "minimize jerks, etc."). Additionally or alternatively, the planning system 250 may leverage dynamic costs to evaluate trajectories or strategies for the autonomous platform based on predictive results for the current operational scenario (e.g., predicted trajectories or strategies leading to interactions between actors, predicted trajectories or strategies leading to interactions between actors and the autonomous platform, etc.). The planning system 250 can rank trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system 250 can select a motion plan (and corresponding trajectory) based on the ranking of multiple candidate trajectories. In some embodiments, the planning system 250 can select the highest-ranking candidate or the highest-ranking feasible candidate.

[0105] Next, the planning system 250 can verify the selected trajectory for one or more constraints before the trajectory is executed by the autonomous platform.

[0106] To assist in motion planning decisions, the planning system 250 may be configured to perform predictive functions. The planning system 250 can predict future states of the environment. This may include predicting future states of other actors in the environment. In some embodiments, the planning system 250 can predict future states based on current or past states (e.g., developed or maintained by the perception system 230). In some embodiments, the future state may be or include the predicted trajectory (e.g., position over time) of an object in the environment, such as another actor. In some embodiments, one or more of the future states may include one or more probabilities associated with it (e.g., marginal probabilities, conditional probabilities). For example, one or more probabilities may include one or more probabilities conditioned on a strategy or trajectory option available to the autonomous platform. Additionally or alternatively, probabilities may include probabilities conditioned on a trajectory option available to one or more other actors.

[0107] In some embodiments, the planning system 250 may perform interactive predictions. The planning system 250 may determine a motion plan for the autonomous platform by understanding how the predicted future state of the environment may be affected by the execution of one or more candidate motion plans. For example, referring again to Figure 1, the autonomous platform 110 may determine candidate motion plans corresponding to a set of platform trajectories 112A-C corresponding to the first actor trajectories 122A-C for the first actor 120, trajectory 132 for the second actor 130, and trajectory 142 for the third actor 140 (for example, each trajectory correspondence is shown in a matching line style). For example, the autonomous platform 110 may predict (for example, using the autonomous system 200) that platform trajectory 112A, which moves the autonomous platform 110 more quickly in the area in front of the first actor 120, is associated with the first actor 120 reducing its forward speed according to the first actor trajectory 122A and making a more rapid concession to the autonomous platform 110. Additionally or alternatively, the autonomous platform 110 can predict that a platform trajectory 112B that smoothly moves the autonomous platform 110 into the area in front of the first actor 120 is associated with the first actor 120 slightly reducing its speed according to the first actor trajectory 122B and slowly making concessions to the autonomous platform 110. Additionally or alternatively, the autonomous platform 110 can predict that a platform trajectory 112C that maintains alignment parallel to the first actor 120 is associated with the first actor 120 not making any concessions to the autonomous platform 110 by any distance according to the first actor trajectory 122C. Based on comparing the predicted scenarios with a desired set of results (for example, by scoring the scenarios based on cost or compensation), the planning system 250 can select a motion plan (and associated trajectories) that takes into account the interaction between the autonomous platform and the environment 100. In this way, for example, the autonomous platform 110 can interleave the prediction and motion planning functions.

[0108] To implement the selected motion plan, the autonomous system 200 may include a control system 260 (e.g., a vehicle control system). Generally, the control system 260 may provide an interface between the autonomous system 200 and the platform control device 212 to implement the strategy and motion plan generated by the planning system 250. For example, the control system 260 can implement the selected motion plan / trajectory to control the movement of the autonomous platform through the environment by following the selected trajectory (e.g., a trajectory in which waypoints are included). For example, the control system 260 can translate the motion plan into commands to the appropriate platform control device 212 (e.g., acceleration control, brake control, steering control, etc.). For example, the control system 260 can translate the selected motion plan into commands such as adjusting the steering component (e.g., steering angle) by a certain number, applying a braking force of a certain size, or increasing / decreasing speed. In some embodiments, the control system 260 may communicate with the platform control unit 212 via a communication channel that includes, for example, one or more data buses (e.g., a controller area network (CAN)), an onboard diagnostic connector (e.g., OBD-II), or a combination of wired or wireless links. The platform control unit 212 may send or receive data, messages, signals, etc. to or from the autonomous system 200 via the communication channel (and vice versa).

[0109] The autonomous system 200 may receive support signals from the remote support system 270 via the communication interface 206. The remote support system 270 may communicate with the autonomous system 200 via a network (for example, as a remote system 160 via network 170). In some embodiments, the autonomous system 200 may initiate a communication session with the remote support system 270. For example, the autonomous system 200 may initiate a session based on or in response to a trigger. In some embodiments, the trigger may be a warning, an error signal, a map feature, a request, a location, a traffic condition, a road condition, etc.

[0110] After the session has started, the autonomous system 200 may provide context data to the remote assistance system 270. Context data may include sensor data 204 and autonomous platform state data. For example, context data may include live camera feeds from the autonomous platform's cameras and the current speed of the autonomous platform. The operator of the remote assistance system 270 (e.g., a human operator) can use the context data to select assistance signals. Assistance signals may provide values ​​or adjustments for various operating parameters or characteristics of the autonomous system 200. For example, assistance signals may include waypoints (e.g., obstacle routing, lane changes), speed or acceleration profiles (e.g., speed limits), relative motion instructions (e.g., convoy formation), operating characteristics (e.g., use of auxiliary systems, reduction of energy processing modes), or other signals to assist the autonomous system 200.

[0111] The autonomous system 200 can use support signals input to one or more autonomous subsystems to perform autonomous functions. For example, the planning subsystem 250 may receive support signals as inputs for generating a motion plan. For example, the support signals may include constraints for generating the motion plan. Additionally or alternatively, the support signals may include costs or compensatory adjustments that influence the motion plan by the planning subsystem 250. Additionally or alternatively, the support signals may be considered as suggestive inputs that may be taken into account by the autonomous system 200 along with other received data (e.g., sensor inputs).

[0112] The autonomous system 200 may be platform-independent, and the control system 260 may provide control commands to the platform control device 212 for various platforms for autonomous mobility (e.g., multiple different autonomous platforms equipped with autonomous control systems). This may include various different types of autonomous vehicles (e.g., sedans, vans, SUVs, trucks, electric vehicles, combustion-powered vehicles, etc.) from various different manufacturers / developers operating in various different environments and, in some embodiments, providing one or more vehicle services.

[0113] For example, referring to Figure 3a, the operating environment may include a dense environment 300. The autonomous platform may include an autonomous vehicle 310 controlled by the autonomous system 200. In some embodiments, the autonomous vehicle 310 may be configured for maneuverability in a dense environment, for example, having a set wheelbase or other specifications. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo or passengers. In some embodiments, the autonomous vehicle 310 (e.g., a passenger van, shuttle, bus) may be configured to transport a large number of passengers. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo, for example, large cargo (e.g., a truck, box van, step van, etc.) or small cargo (e.g., food, personal packaging, etc.).

[0114] Referring to Figure 3b, a selected overhead view 302 of the dense environment 300 is shown superimposed on an exemplary trip / service between a first location 304 and a second location 306. The exemplary trip / service can be assigned to an autonomous vehicle 320 by, for example, a remote computing system. The autonomous vehicle 320 may be, for example, the same type of vehicle as autonomous vehicle 310. The exemplary trip / service may include transporting passengers or cargo between the first location 304 and the second location 306. In some embodiments, the exemplary trip / service may include traveling to or through one or more intermediate locations to load or unload passengers or cargo. In some embodiments, the exemplary trip / service may be pre-booked (for example, for regular operation by transport schedule, etc.). In some embodiments, the exemplary trip / service may be on-demand (for example, by request or for the performance of a taxi, ride-sharing, ride-hailing, courier, delivery service, etc.).

[0115] Referring to Figure 3c, in other examples, the operating environment may include an open-type mobile path environment 330. The autonomous platform may include an autonomous vehicle 350 controlled by the autonomous system 200. This may include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle 350 may be configured for high-load transport, such as long-distance, high-load transport (e.g., transport of large quantities of cargo or other goods or passengers). For example, the autonomous vehicle 350 may include one or more cargo platform attachments, such as a trailer 352. Although shown as a towed attachment in Figure 3c, in some embodiments, one or more cargo platforms (e.g., box van, step van, etc.) (e.g., mounted on a chassis) may be integrated into the autonomous vehicle 350.

[0116] Referring to Figure 3d, a selected overhead view of the open travel path environment 330 is shown, including travel paths 332, interchanges 334, transfer hubs 336 and 338, access travel paths 340, and locations 342, 344. In some embodiments, an autonomous vehicle (e.g., autonomous vehicle 310 or autonomous vehicle 350) may be assigned an exemplary trip / service to operate one or more travel paths 332 (selectively connected by interchanges 334) to transport cargo between transfer hubs 336 and 338. For example, in some embodiments, the exemplary trip / service may include cargo delivery / transportation services, such as cargo delivery / transportation services. The exemplary trip / service may be assigned by a remote computing system. In some embodiments, transfer hub 336 may be a cargo origin (e.g., warehouse, wholesale store, facility, etc.), and transfer hub 338 may be a cargo destination (e.g., retail store, etc.). However, in some embodiments, the transfer hub 336 may be an intermediate point along the final journey between each origin and each destination of the cargo. For example, the origin of the cargo may be located at location 342 along the access travel route 340. Thus, the cargo can be transported to the transfer hub 336 for staging (e.g., by a human-driven vehicle, an autonomous vehicle 310, etc.). At the transfer hub 336, various cargo can be grouped or staged for long-distance transport along the travel route 332.

[0117] In some embodiments of the exemplary trip / service, a group of staged cargo items may be loaded onto an autonomous vehicle (e.g., autonomous vehicle 350) for transport to one or more other transit hubs, such as transit hub 338. For example, although not shown, it should be understood that the open travel route environment 330 may include more transit hubs than transit hubs 336 and 338, and may include more travel routes 332 interconnected to more interchanges 334. A simplified map is presented here for clarity purposes only. In some embodiments, one or more cargo items transported to transit hub 338 may be distributed to one or more local destinations along a travel route 340 to access location 344 (e.g., by a human-driven vehicle, autonomous vehicle 310, etc.). In some embodiments, the exemplary trip / service may be pre-booked (e.g., for regular operations such as a transport schedule). In some embodiments, the exemplary trip / service may be on-demand (e.g., to provide chartered passenger transport or cargo delivery services, or upon request for such services).

[0118] This disclosure relates to a predictive technique that helps improve the planning performance of an autonomous platform 110, such as an autonomous vehicle (e.g., an autonomous vehicle 310 or 350) that is at least partially controlled using an autonomous system 200.

[0119] Figure 4 is a block diagram of a system 400 configured to initiate the movement of an autonomous platform according to some embodiments of the present disclosure. One or more parts of system 400 may be included in and / or associated with a planning system of the autonomous platform (e.g., planning system 250 in Figure 2). System 400 may include a context memory 405. The context memory 405 may include a cache memory that stores context data processed by a prediction system 410, a strategy enumeration system 415, a strategy ranking system 420, a trajectory optimization system 425, and a plan arbitration system 430, which can determine how the autonomous platform interacts with the environment and within that environment. The context data may include map data (e.g., map data 210 in Figure 2), sensor data (e.g., sensor data 204 in Figure 2), and / or state data representing one or more current, past, and / or predicted future states of actors in the environment of the autonomous vehicle.

[0120] The prediction system 410 may include a machine learning prediction model capable of processing contextual data to generate discrete intentions and continuous motion outputs that can be used to perform one or more parts of a motion planning process. The discrete intentions and continuous motion outputs may include one or more predicted targets 435, predicted interactions 440, and / or continuous trajectories 445 for each individual actor in the environment.

[0121] The strategy enumeration system 415 may determine multiple feasible strategies for the autonomous platform based on context data from the context memory 405. Each individual strategy may include multiple discrete decisions that the autonomous platform may make within the operating time of the environment. These discrete decisions may include interpretable decisions such as concessions to actors, merging behind actors, and / or any other interpretable movement path decisions. In some embodiments, the strategy enumeration system 415 may determine multiple feasible strategies for the autonomous platform based on anticipated goals 435 and / or anticipated interactions 440 with respect to actors in the environment.

[0122] The strategy ranking system 420 may process feasible strategies and contextual data and determine the ranking of one or more feasible strategies. The strategy ranking system 420 may determine the ranking of feasible strategies by one or more costs associated with those strategies. In some embodiments, the strategy ranking system 420 may include one or more machine learning models trained on expert human driving data. One or more machine learning models may be trained to determine the costs corresponding to the difference between individual strategies and expert human driver strategies in the same driving scenario. In some embodiments, the strategy ranking system 420 may consider predicted goals 435 and / or predicted interactions 440 for actors in the environment when determining the ranking of feasible strategies.

[0123] The trajectory optimization system 425 can identify the optimal trajectory for each of the strategies identified based on contextual data and one or more cost functions. For example, the cost function may include a static cost function that encodes one or more desired driving actions, such as avoiding lane boundaries, maintaining near the center of the lane, avoiding acceleration and / or jerks, and avoiding steering jerks. Additionally or alternatively, the cost function may include a dynamic cost function that can evaluate dynamic constraints. For example, the dynamic cost function can evaluate predicted targets 435, predicted interactions 440, and / or continuous trajectories 445 for actors in the environment.

[0124] The planning arbitration system 430 can select the optimal trajectory and strategy pair for execution by the autonomous platform. To this end, the planning arbitration system 430 can reject one or more trajectories that result in interference with other actors / objects, lane boundary violations, etc. The planning arbitration system 430 can select the optimal trajectory and strategy pair from among the non-rejected trajectories that optimize (e.g., minimize) the total cost evaluated by the static and / or dynamic cost functions described herein. In some embodiments, the planning arbitration system 430 can select the optimal trajectory and strategy pair based on predicted objectives 435 for actors in the environment.

[0125] The autonomous platform may initiate movement based on a selected optimal trajectory and strategy pair. In this way, the autonomous platform's movement may be initiated based on a predicted target 435, a predicted interaction 440, and / or a continuous trajectory 445 output by the prediction system 410.

[0126] Figure 5 is an exemplary environment 500 that can be modeled by data provided as input to a prediction system to determine predictable targets, predictable interactions, and / or continuous trajectories according to some embodiments of the present disclosure. More specifically, the data representing the environment 500 in Figure 5 may be provided as input to an exemplary prediction system 700 in Figure 7, which is trained to encode the data into a graph neural network (GNN) 600, as shown in Figure 6.

[0127] Environment 500 may include multiple actors 510A-B. Actors can be dynamic objects, such as vehicles, pedestrians, and / or any other objects that change their position within Environment 500 over time.

[0128] The prediction system 700 in Figure 7 may acquire state data 702 associated with each of several actors in the environment (e.g., actors 510A-B in environment 500 in Figure 5). The state data 702 may include physical characteristics (e.g., bounding box, shape, etc.) and / or multiple actor states associated with each individual actor. Multiple actor states may indicate one or more positions, one or more velocities, and / or one or more directions of travel for each individual actor at the present time and one or more past time points. For example, multiple actor states may include one or more past states 525A-B and / or current states 520A-B for each of the multiple actors 510A-B in environment 500. Current states 520A-B may include position, velocity, acceleration, direction of travel, etc. at the present time. Past states 525A-B may include position, velocity, acceleration, direction of travel, etc. at past time points prior to the present time. Past time points may be within a range of one or more past time points. The past time range may include one or more seconds, minutes, etc., prior to the present time. In some embodiments, past states 525A-B may include multiple past positions, velocities, accelerations, directions of travel, etc., for individual actors within a 5-second time range prior to the present time.

[0129] Additionally or alternatively, the prediction system 700 in Figure 7 may acquire map data 704 related to the environment (e.g., map data for environment 500 in Figure 5). The map data 704 may show multiple lanes in environment 500 and / or map features for those multiple lanes. For example, the map data 704 may include a lane graph describing the multiple lanes and / or right of way information for those multiple lanes. Additionally or alternatively, the map features in the map data 704 may identify lane centerline locations (e.g., nominal waypoints), lane boundary locations, lane boundary types (e.g., physical barriers, solid lines, dashed lines, etc.), speed limits and / or any other physical or traffic information for the multiple lanes in environment 500.

[0130] Map data 704 can define nominal routes for each of multiple lanes. A nominal route can be defined by multiple waypoints along the centerline of an individual lane and / or within a critical distance of that centerline. For example, multiple map features may include multiple waypoints for each nominal route defined in map data 704. For each waypoint, the map feature can identify the position of the lane at that waypoint, the speed limit, and / or any other optional attributes.

[0131] A part of the prediction system 700 (for example, one or more graph generation layers 712 of the encoder 710) may be configured to determine one or more potential targets for an actor from state data 702 and map data 704 provided as input to a machine learning prediction model 706. The potential targets may indicate the actor's intended destination within the environment 500 and the nominal route to that destination. For example, a potential target may represent a location within the environment 500 that the actor may occupy in the future. As an example, a potential target 505A may represent a location within the environment 500 that the first actor 510A would occupy if the first actor 510A decided to turn right. The potential targets may coincide with or be determined by lanes available to the actor in the immediate vicinity of the actor.

[0132] Each potential objective may include a target path and a target destination. The target destination may include future destination locations for individual actors within the environment 500. Future destination locations may be based on discrete decisions predicted for individual actors. For example, a future destination location may be identified based on the prediction that an individual actor will turn right, while other future destination locations may be identified based on the prediction that an individual actor will continue straight, for example. The target path may include one or more waypoints along the path from an individual actor's current location to its individual future destination location.

[0133] The prediction system 700 can determine potential targets for each actor within the environment 500 based on contextual data about the environment 500 (e.g., available lanes for vehicles). In some embodiments, the target route for a potential target may include a nominal route defined in map data 704. For example, a potential target may indicate one of the actor's intended destinations and the nominal route to that intended destination. The map data 704 may include multiple map features associated with each potential target. The map features can identify multiple waypoints on the individual nominal routes to the potential target. In this way, a potential target can define the spatial aspects of the actor's intended future actions (e.g., turning right, going straight) based on the map data 704 corresponding to the environment 500.

[0134] The prediction system 700 can determine a number of first potential targets 505A to D for the first actor 510A based on the first actor 510A's current state (e.g., location, direction of travel, etc.) and map data. For example, the prediction system 700 can identify each nominal route defined by map data including waypoints within a search radius 530 of the first actor 510A's current state 520A.

[0135] As an example, multiple first potential targets 505A-D for the first actor 510A may include four potential targets 505A-D based on their position within the environment 500. The first target 505A may include the nominal route and destination position when the first actor 510A intends to turn right. The second target 505B may include the nominal route and destination position when the first actor 510A continues to go straight. The third target 505C may include the nominal route and destination position when the first actor 510A intends to change lanes to the left. The fourth target 505D may include the nominal route and destination position when the first actor 510A intends to turn left.

[0136] The prediction system 700 may determine a number of second potential targets 515A to D for the second actor 510B in the environment 500 based on the second actor 510B's current state (e.g., position, direction of travel, etc.) and map data. The number of second potential targets 515A to D for the second actor 510B may include four different potential targets 515A to D based on the second actor's position in the environment 500. The first target 515A may include the nominal route and destination position if the second actor 510B intends to turn left. The second target 515B may include the nominal route and destination position if the second actor 510B continues to go straight. The third target 515C may include the nominal route and destination position if the second actor 510B intends to change lanes to the right. The fourth target 515D may include the nominal route and destination position if the second actor 510B intends to turn right.

[0137] In some embodiments, the prediction system 700 may determine a goal-free mode 535 to indicate that an individual actor (e.g., a first actor 510A) may not intend to make a decision based on any of the identified potential goals 505A-D (e.g., by deviating from the road or stopping).

[0138] The prediction system 700 may be configured to predict the probability that each individual actor will follow their respective goals and / or no-goal modes. For example, the prediction system 700 may process state data 702, map data 704, and multiple potential goals for each actor in the environment 500 as one or more parts of a machine learning prediction model 706 to determine predicted goals for individual actors of multiple actors 510A-B. As an example, the machine learning prediction model 706 may determine the probability of each potential goal for individual actors. The predicted goals may include the individual potential goals with the highest probability.

[0139] As will be described in more detail herein, the machine learning prediction model 706 can also determine at least one predicted interaction between individual actors of a plurality of actors 510A-B and other actors based on the predicted goal, and at least one sequential trajectory for individual actors based on the predicted goal.

[0140] Figure 6 is a graph representation of a graph neural network (GNN) 600 generated using a machine learning prediction model (e.g., machine learning prediction model 706 in Figure 7) according to some embodiments of the present disclosure. The graph neural network 600 may include multiple nodes and multiple edges. The multiple nodes and multiple edges may be based on multiple actors and multiple goals of the environment 500 in Figure 5. The multiple goals and multiple edges represented in the GNN 600 in Figure 6 may be generated within the graph generation layer 712 of the machine learning prediction model 706 in Figure 7.

[0141] For example, multiple nodes of GNN600 may include multiple actor nodes 610A to B, each corresponding individually to one of the multiple actors 510A to B within the environment 500. Multiple actor nodes 610A to B may include at least one actor node for each actor within the environment 500. As an example, multiple actor nodes 610A to B may include a first actor node 610A corresponding to the first actor 510A in the environment 500 and / or a second actor node 610B corresponding to the second actor 510B in the environment 500.

[0142] The multiple nodes of the GNN600 may further include multiple target nodes that individually correspond to each of the multiple potential targets identified for actors 510A-B. The multiple target nodes may include at least one target node for each target identified for each actor in the environment 500. For example, the multiple target nodes may include multiple first target nodes 605A-D corresponding to the first actor node 610A (for example, for the first actor 510A) and multiple second target nodes 615A-D corresponding to the second actor node 610B (for example, for the second actor 610B).

[0143] Multiple edges in GNN600 may include one or more actor-target edges that individually connect individual actor nodes and individual target nodes. An actor-target edge may include at least one actor-target edge between an individual actor and each target identified for each individual actor in the environment. For example, multiple actor-target edges may include multiple first actor-target edges 620A-D corresponding to the first actor node 610A and the multiple first target nodes 605A-D (for example, for multiple potential targets 505A-D identified for a first actor 510A). Additionally or alternatively, multiple actor-target edges may include multiple second actor-target nodes 625A-D corresponding to the second actor node 610B and the multiple second target nodes 615A-D (for example, for multiple potential targets 515A-D identified for a second actor 510B).

[0144] Multiple edges may further include one or more target-target edges 630A~F that individually connect at least two target nodes from among multiple target nodes. A target-target edge can represent a relationship between at least two actors in the environment and two target nodes individually associated with them. For example, target-target edges 630A~F include (1) a first target-target edge 630A showing the relationship between a first target node 605A associated with a first actor 510A and a first target node 615A associated with a second actor 510B, (2) a second target-target edge 630B showing the relationship between a first target node 615A associated with a second actor 510B and a third target node 605C associated with a first actor 510A, and (3) a third target-target edge showing the relationship between a first target node 615A associated with a second actor 510B and a second target node 605B associated with a first actor 510A. The edge 630C may include (4) a fourth target-target edge 630D showing the relationship between a fourth target node 615D related to the second actor 510B and a fourth target node 605D related to the first actor 510A, (5) a fifth target-target edge 630E showing the relationship between a fourth target node 605D related to the first actor 510A and a second target node 615B related to the second actor 510B, and (6) a sixth target-target edge 630F showing the relationship between a fourth target node 605D related to the first actor 510A and a third target node 615C related to the second actor 510B.

[0145] The relationship between at least two goal nodes connected by individual goal-goal edges may be based on a potential interaction between two actors if the two actors move according to at least two goals represented by the connected goal nodes.

[0146] The nodes and edges of the graph neural network 600 may contain multiple feature embeddings. These multiple feature embeddings can encode different characteristics based on the corresponding nodes and / or edges of the graph neural network 600. For example, the multiple feature embeddings may include multiple actor feature embeddings corresponding to multiple actor nodes 610A-B, multiple target feature embeddings corresponding to multiple target nodes, multiple actor-target feature embeddings corresponding to multiple actor-target edges, and one or more target-target feature embeddings corresponding to one or more target-target edges 630A-F.

[0147] Referring again to Figure 7, the prediction system 700 can process state data 702 and map data 704 using the encoder portion 710 of the machine learning prediction model 706. The encoder portion 710 may include multiple encoding layers. Multiple encoding layers may be shared among multiple encoding tasks and may include multiple graph generation layers 712 and multiple message transfer layers 714. The multiple graph generation layers 712 are for determining nodes and edges in the GNN (e.g., GNN600 in Figure 6) according to multiple potential goals determined for actors in the environment and for generating potential representations for each node and edge. Potential representations may take the form of multiple embeddings, including target feature embeddings, actor feature embeddings, target-target feature embeddings, and actor-target feature embeddings. While Figure 7 depicts encoding for all these feature embeddings being performed within a single encoder portion 710, it should be understood that separate encoder networks or models (e.g., target feature model, actor feature model, target-target feature model, and actor-target feature model) may be provided to generate and update different types of embeddings.

[0148] Once the latent representations of nodes and edges are encoded by the graph generation layer 712 of the encoder portion 710, one or more message transfer layers 714 (e.g., layers similar to the convolutional layers in a CNN) are used to learn the updated graph embeddings and generate a compact feature representation 720.

[0149] More specifically, one or more message transfer layers 714 of a machine learning prediction model may perform one or more message transfer rounds to share information between multiple nodes and multiple edges of a graph neural network to perform scene-level inference. For example, a machine learning prediction model 706 may perform one or more message transfer rounds via a series of message transfers 714 to generate multiple updated feature embeddings based on information from neighboring feature embeddings. During each message transfer round, each feature embedding can be updated based on information encoded within each neighboring feature embedding. In other words, the node / edge-to-node message transfers of the GNN implemented within the message transfer layers 714 help capture dependencies within the graph by causing nodes in the graph to aggregate messages from neighboring nodes. For example, for a given target node represented by one or more feature embeddings, each message transfer layer is responsible for updating the feature embedding of a given target node so that it includes data related to the node's previous representation and data related to the node's nearest neighbor node in the graph (e.g., GNN600). Each iteration updating the feature embeddings may correspond to one of the multiple message transfer layers 714. In each iteration, the feature embedding data is shared one hop or one step along the graph. Thus, multiple message forwarding layers 714 can be configured to include at least as many layers as there are path distances in the graph, where path distance is defined as the number of discrete steps between nodes in the graph.

[0150] In some embodiments, the number of message transfer rounds corresponding to multiple layers within multiple message transfer layers 714 can be determined statically and / or dynamically based on one or more characteristics of the environment. For example, the number of message transfer rounds may be based on the number of actors in the environment, the number of potential targets for each actor in the environment, the number of potential interactions between actors in the environment, etc. In some embodiments, the number of message transfer rounds can be determined by the number of nodes, edges, and / or feature embeddings in the graph neural network. For example, the number of message transfer rounds can be determined so that individual feature embeddings can be updated based on information about feature embeddings corresponding to the furthest nodes and / or edges of the graph neural network.

[0151] A machine learning prediction model may include various components for message forwarding. A machine learning prediction model can leverage a goal-target edge model configured to update goal-target feature embeddings based on combinations of features of adjacent goal nodes. For example, a goal-target edge model can be trained to update goal-target feature embedding 830 based on features transmitted between goal nodes 805A and 805C.

[0152] Machine learning prediction models can leverage actor-target edge models for message forwarding. An actor-target edge model can be configured to update actor-target feature embeddings based on combinations of features between adjacent actor and target nodes. For example, an actor-target model can be trained to update actor-target feature embedding 825 based on features transmitted between the embeddings of actor node 810A and target node 805A, based on features transmitted between nodes.

[0153] Machine learning prediction models can leverage target node models for message forwarding. A target node model can be configured to update embeddings associated with a target node based on features of one or more other target nodes. For example, a target node model can update feature embeddings associated with a target node based on an aggregate of embedding updates from other target-to-target feature embeddings and actor-to-target feature embeddings.

[0154] Machine learning prediction models can leverage actor-node models for message forwarding. An actor-node model can be configured to update embeddings associated with an actor node based on features of one or more other actor nodes. For example, an actor-node model can update actor feature embedding 850 associated with actor node 810A based on features propagated from other actor nodes.

[0155] The message transmission layer can be trained with three decoder heads using the same data as each head (for example, in the case of a target probability head, labeled ground truth target paths determined by mean cross-track distances). In some embodiments, the entire machine learning predictive model can be trained in an end-to-end manner. This training can be performed using the systems and hardware described herein.

[0156] Referring further to Figure 7, the feature representation 720 may correspond to, or include, any suitable representation for computationally describing multiple updated feature embeddings, such as feature vectors, but is not limited to these. The feature representation 720 may include, or otherwise describe, multiple learned feature embeddings, including target feature embeddings, actor feature embeddings, target-target feature embeddings, and actor-target feature embeddings, as described herein.

[0157] The feature representation 720 can be decoded using the decoder 730 to provide the output of the machine learning prediction model 706. The decoder 730 may be a trained model trained using a process similar to that described in relation to Figures 16 and 17. The decoder 730 may enable the machine learning prediction model 706 to determine various types of outputs. The feature representation 720 can then be provided as input to the decoder portion 730 of the machine learning prediction model 706. The decoder portion 730 may include multiple decoding layers. The decoder portion 730 can be trained to make predictions from node and edge embeddings provided within the feature representation 720. More specifically, the decoder model 730 may be configured to generate predictions corresponding to (i) one or more target probabilities 740, (ii) one or more interaction probabilities 750, and / or (iii) one or more continuous trajectories 760.

[0158] The encoder portion 710 of the machine learning prediction model 706 is configured to generate multiple feature embeddings 720 corresponding to multiple actors and multiple potential goals. For example, the encoder portion 710 of the machine learning prediction model 706 can encode multiple state and / or map features from state data 702 and map data 704 for individual feature embeddings within the multiple feature embeddings 720. The encoder portion 710 may include a differentiable nonlinear transformation configured to transform raw input (e.g., state / map data) into potential feature embeddings. In some embodiments, the encoder portion 710 may include multiple two-layer multilayer perceptrons as part of the framework within the graph generation layer 712.

[0159] The encoder section 710 can process the state data 702 to generate actor feature embeddings for individual actor nodes. The actor feature embeddings may be based on multiple actor states (e.g., current, past states, etc.) associated with individual actors of multiple actors 510A-B. For example, the actor feature embeddings can encode information related to the current and / or past position, direction of travel, footprint, etc., of individual actors in the actor coordinate frame.

[0160] Each actor node may contain individual actor feature embeddings. For example, the first actor node 610A may contain actor feature embeddings that encode the current and / or past location, direction of travel, footprint, etc., of the first actor. Additionally or alternatively, the second actor node 610B may contain actor feature embeddings that encode the current and / or past location, direction of travel, footprint, etc., of the second actor.

[0161] The encoder unit 710 can process the map data 704 to generate target feature embeddings for individual target nodes. Target feature embeddings can be generated by processing multiple map features related to the destination locations of individual targets. For example, target feature embeddings can encode information related to the destination locations of individual targets in the environment.

[0162] Each target node may contain individual target feature embeddings. For example, the first target node 605A for the first actor 510A may contain a target feature embedding related to the location of the target destination of the first target 505A. The second target node 605B for the first actor 510A may contain a target feature embedding related to the location of the target destination of the second target 505B. The first target node 615A for the second actor 510B may contain a target feature embedding related to the location of the target destination of the first target 515A. The second target node 615B for the second actor 510B may contain a target feature embedding related to the location of the target destination for the second target 515B, and so on.

[0163] The encoder section 710 can process state data 702 and map data 704 to generate actor-target feature embeddings for individual actor-target edges. The actor-target feature embeddings can encode information related to multiple actor states for an individual target. For example, multiple actor states for a first target may be multiple actor states in a path tangent frame.

[0164] Each actor-target edge may include individual actor-target feature embeddings. For example, the first actor-target edge 620A may include individual actor-target feature embeddings that encode current and / or past actor state information related to the first actor 510A corresponding to the first actor node 610A, for the first target 505A corresponding to the first target node 605A, and the first actor-target edge 625A may include individual actor-target feature embeddings that encode current and / or past actor state information related to the first actor 510B corresponding to the second actor node 610B, for the first target 515A corresponding to the first target node 615A.

[0165] The encoder section 710 can process state data 702 and map data 704 to generate goal-target feature embeddings for individual goal-target edges. Each goal-target feature embedding can encode information related to multiple actor states for an individual actor (e.g., first actor 510A) for other actors (e.g., second actor 510B) within a group of actors 510A~B. For example, each goal-target feature embedding can encode state information for individual actors within the frame of other actors.

[0166] Furthermore, the goal-goal feature embedding can further encode one or more distances between an actor and a shared space associated with at least two goals connected by individual goal-goal edges. For example, the distances may include a first distance between an individual actor and the shared space, and a first distance between other actors and the shared space.

[0167] Each goal-target edge may contain individual goal-target feature embeddings. For example, the first goal-target edge 630A between the first goal node 605A associated with the first actor node 610A and the first goal node 615A associated with the second actor node 610B may contain a goal-target feature embedding that encodes state information for the first actor 510A, which corresponds to the first actor node 610A, with respect to the second actor 510B, which corresponds to the second actor node 610B. The first goal-target feature embedding can also encode the distance between the first actor 510A and the shared space 710, and the distance between the second actor 510B and the shared space 710.

[0168] As described above, the machine learning prediction model 706 may use the decoder portion 730 to process multiple feature embeddings provided to the feature representation 720 and determine the predicted target 740, predicted interaction 750, and / or continuous trajectory 760 for each actor in the environment.

[0169] In some embodiments, the technology of the present disclosure may evaluate a scene and make related predictions by representing an autonomous vehicle, including (or running) a graph neural network, using an onboard computing system as a node in the graph neural network. In this way, the computing system of the autonomous vehicle can evaluate the autonomous vehicle as an actor in the scene.

[0170] To this end, the computing system may acquire state data related to the autonomous vehicle and map data representing multiple lanes in the environment. State data related to the autonomous vehicle may be generated using one or more of the vehicle's onboard systems. For example, state data may be associated with the autonomous vehicle's position estimation system (e.g., position estimation system 230 in Figure 2) or an inertial measurement unit (IMU). For example, position information encoded in the coded state may be generated by the position estimation system. Additionally or alternatively, information such as velocity, acceleration, and direction of travel encoded in the state data may be generated by the IMU.

[0171] As similarly described here, the computing system may determine at least one potential target for the autonomous vehicle based on state data and map data. The potential target may include the location of a potential destination in the environment and the target path to the location of the potential destination for the autonomous vehicle.

[0172] A computing system may use a machine learning predictive model to process state data, map data, and at least one potential target to determine (i) a predicted target for the autonomous vehicle, (ii) a predicted interaction between the autonomous vehicle and other actors based on the predicted target, and (iii) a continuous trajectory of the autonomous vehicle based on the predicted target. This may include, for example, representing the autonomous vehicle as a node in a graph neural network. In this way, the autonomous vehicle may use the techniques described herein to predict potential interactions between the autonomous vehicle and other actors in the vehicle environment and plan its movements accordingly.

[0173] Figure 8 is an exemplary diagram of the data flow within System 800 according to some embodiments of the present disclosure. System 800 may correspond to the prediction system 700 of Figure 7, which can be used to generate a graph neural network comprising a plurality of actor nodes 810A-C, a plurality of target nodes 805A-C, a plurality of actor-target edges 815A-C, and a plurality of target-target edges 820A-C. Each of the plurality of nodes and edges may be associated with a separate feature embedding. For example, actor-target edge 815A may include actor-target feature embedding 825, target-target edge 820A may include target-target feature embedding 830, and actor node 810A may include actor feature embedding 850. These feature embeddings 825, 830, and 850 may be included as part of a feature representation transmitted to decoder 860. Decoder 860 can be trained to receive the feature representation as input and generate a plurality of predictions corresponding to target probabilities 835, continuous trajectories 840, and interaction probabilities 845. Decoder 860 may correspond to decoder 730 as described with reference to Figure 7.

[0174] The machine learning prediction model 706 may, for example, determine target probabilities 835, interaction probabilities 845, and / or continuous trajectories 840 for individual actors in the environment based on multiple updated feature embeddings.

[0175] The machine learning prediction model 706 can predict a target probability 835 for each actor. The target probability 835 can indicate whether a particular actor will follow each of the available potential target paths or not follow any of the target paths. The target probability 835 can take the form of a probability distribution for potential target paths, including cases where there are no target paths. To determine the target probability 835, the machine learning prediction model can process actor feature embeddings and actor-target feature embeddings. For example, the machine learning prediction model 706 may process at least one of actor feature embeddings 850 or actor-target feature embeddings 825 to determine the target probability 835 for each individual actor corresponding to actor node 810A. In this process, the machine learning prediction model 706 may determine at least one of a target-based score or a no-target score 855 for each individual actor. The target-based score can enable the machine learning prediction model 706 to determine the probability that an individual actor moves according to an individual potential target and output a probabilistic value associated with such a move. The score 855 without a target may include a probabilistic target value indicating the probability that an individual actor will not move according to any identified potential target. A machine learning predictive model may use the target-based score and / or the score 855 without a target to determine the target probability 835 for individual actors associated with actor node 810A. For example, the target probability 835 may be based on the probability associated with each identified potential target for the individual actor and / or the probability that the actor will not move according to any of the identified potential targets. For example, the target probability 835 may include the individual potential targets with the highest probability.

[0176] The machine learning prediction model 706 may determine the interaction probability 845 for each pair of overlapping target paths for different actors. The interaction probability 845 can indicate the likelihood that one actor will make concessions to another actor, make reverse concessions, or ignore other actors, under the assumption that the actors follow the overlapping target paths. The interaction probability 845 for overlapping target paths for a particular pair can be expressed as a probability distribution for such interaction types. For example, the machine learning prediction model 706 may process the target-target feature embedding 830 and, based on the predicted targets corresponding to target nodes 805A and 805C, determine the interaction probability 845 for overlapping target paths between the actor corresponding to actor node 810A and the other actor corresponding to actor node 810C.

[0177] The machine learning prediction model 706 may determine a sequence of trajectories 840 for each individual actor based on multiple feature embeddings and the predicted goals for each individual actor. For example, the machine learning prediction model 706 may determine a sequence of trajectories 840 for each individual actor associated with actor node 810A based on the predicted goals 835 and actor-goal feature embeddings 825. The sequence of trajectories 840 may include future trajectories predicted for each individual actor, conditionally with respect to a specific goal.

[0178] In one example, Figure 9 is an exemplary Figure 900 which includes a sequence of trajectories that can be transformed to be input into a machine learning prediction system by certain embodiments of the present disclosure. Exemplary Figure 900 includes an environment 500 having a first actor 510A and a second actor 510B. Exemplary Figure 900 includes four sequence of trajectories 905A–D conditioned according to each potential target identified for the first actor 510A. Each sequence of trajectories 905A–D may include a number of waypoints representing the potential future positions of the first actor 510A if the first actor 510A decides to move according to the identified target.

[0179] The waypoints of each continuous trajectory 905A to D may be conditioned according to the individual objectives 505A to D in Figure 5. For example, continuous trajectory 905A may include multiple waypoints conditioned according to the objective path of potential objective 505A. Continuous trajectory 905B may include multiple waypoints conditioned according to the objective path of potential objective 505B. Continuous trajectory 905C may include multiple waypoints conditioned according to the objective path of potential objective 505C. Trajectory 905D may include multiple waypoints conditioned according to the objective path of potential objective 505D.

[0180] Furthermore, a continuous trajectory may include a trajectory 915 without a target. A trajectory without a target can indicate the probability that the first actor 510A does not move according to any of the potential trajectories 905A-D. A trajectory without a target 915 may have a null value. Additionally or alternatively, a continuous trajectory may include one or more waypoints conditioned according to anomalous behavior (e.g., stopping on the shoulder) that may cause the first actor 510A to deviate from the identified potential target.

[0181] Returning to Figure 8, the continuous trajectory 840 may be conditioned according to the target probability 835 and refined based on information encoded by the actor-target feature embedding 825. For example, the continuous trajectory 840 may include a number of waypoints conditioned according to a target path associated with the target probability 835. The positions and / or timings of the waypoints may be offset from the target path based on state data and / or map data encoded by the actor-target embedding 825. In some embodiments, the continuous trajectory 840 may be generated based on updated data encoded by the updated actor-target embedding. In this way, the continuous trajectory 840 may be generated based on information aggregated across each of a number of feature embeddings corresponding to a number of nodes and edges of the graph neural network.

[0182] A machine learning prediction model can output a target probability 835 for each actor in the environment and a continuous trajectory 840 conditioned according to the target probability 835. Actors may include autonomous platforms operating in the environment. In some embodiments, the machine learning prediction model can condition the target probability 835 and / or continuous trajectory 840 based on the known behavior of the autonomous platform.

[0183] Figure 10 is a diagram of the data flow in a system for predicting an actor's intent, which is conditioned according to at least one known decision, as described in some embodiments of the present disclosure. Exemplary Figure 1000 includes a graph neural network generated by the disclosed technique. The graph neural network may include a plurality of actor nodes 810A-C, a plurality of target nodes 805A-C, a plurality of actor-target edges 815A-C, and / or a plurality of target-target edges 820A-C of the graph neural network in Figure 8.

[0184] The machine learning predictive model 706 may receive vehicle motion data 1010 relating to the autonomous platform. The vehicle motion data 1010 can indicate the potential movements of the autonomous platform. For example, the vehicle motion data 1010 can identify discrete motion strategies for the autonomous platform. Discrete motion strategies can indicate interpretable driving decisions such as driving straight, turning, changing lanes, merging into traffic, or stopping. In some embodiments, the discrete motion strategies may coincide with a specific potential goal for at least one actor in the environment (e.g., the autonomous platform).

[0185] The machine learning prediction model 706 can generate conditional target probabilities 1055, conditional interaction probabilities 1060, and / or conditional continuous trajectories 1065 for individual actors conditioned according to the vehicle motion data 1010 by integrating information related to the vehicle motion data 1010 into the latent space of the graph neural network.

[0186] For example, the machine learning prediction model 706 may predict marginal target probabilities, marginal interaction probabilities, and marginal continuous trajectories after at least one round of message transfers between multiple actor nodes 810A-C, multiple target nodes 805A-C, multiple actor-target edges 815A-C, and / or multiple target-target edges 820A-C, and generate multiple updated feature embeddings for multiple nodes and / or edges of the graph neural network.

[0187] The machine learning predictive model 706 may generate conditional feature embeddings 1045 based on the potential movements of the autonomous vehicle. For example, the machine learning predictive model 706 can convert vehicle motion data 1010 into conditional feature embeddings 1045. For example, conditional feature embeddings 1045 can encode state data, map data, and / or other arbitrary data related to the vehicle motion data 1010. The machine learning predictive model may concatenate the conditional feature embeddings 1045 with at least one of several updated feature embeddings. For example, the machine learning predictive model may concatenate the conditional feature embeddings 1045 with the corresponding updated target-target feature embeddings 830. The machine learning predictive model may perform one or more message forwarding rounds with the concatenated feature embeddings to generate multiple conditional feature embeddings that take into account the future movements of the autonomous platform.

[0188] The machine learning predictive model 706 may process multiple conditional feature embeddings (for example, using the decoder 1050) to determine conditional target probabilities 1055, conditional interaction probabilities 1060, and / or conditional continuous trajectories 1065 for individual actors conditioned according to the future movements of the autonomous platform. Herein, as with the decoder described above, the decoder 1050 may be a learned model trained using a process similar to that described herein.

[0189] In some embodiments, the graph neural network may include a typed graph neural network model that includes a different goal-target edge encoder for each interaction type of marginal predictive interaction (e.g., concession, reverse concession, ignore, ignorance, etc.). For example, the typed graph neural network model may include a different concession goal-target encoder for handling goal-target feature embeddings related to marginal predictive interactions of a concession category distribution. The typed graph neural network model may include another reverse concession goal-target encoder for handling goal-target feature embeddings related to marginal predictive interactions of a reverse concession category distribution. The typed graph neural network model may include a different ignore, goal-target encoder for handling goal-target feature embeddings unrelated to marginal predictive interactions. In some embodiments, the typed graph neural network model may include an "ignorance" goal-target encoder for handling goal-target feature embeddings related to marginal predictive interactions where the category distribution is unknown or unknown within a certain confidence threshold. In this way, each goal-target encoder can learn different weights to interpret embeddings corresponding to different interaction types.

[0190] The machine learning predictive model 706 can be trained using one or more machine learning techniques. For example, the machine learning predictive model 706 can be trained using one or more supervised, unsupervised, and reinforcement learning techniques. For example, in some embodiments, the machine learning predictive model 706 can be trained using one or more supervised training techniques based on labeled training data. For example, at least a portion of the labeled training data can be input to the machine learning predictive model 706, and the machine learning predictive model 706 can be trained through back-propagation of errors based on the labels.

[0191] Labeled training data may include data representing multiple real-world environments over time. For example, the data may include map data relating to multiple real-world environments. Additionally or alternatively, the data may include sensor data (e.g., image data, LIDAR data, etc.) relating to multiple real-world environments previously recorded by one or more sensors of the autonomous platform. For example, sensor data may describe one or more traffic sequences representing the training environment and / or one or more actors within the training environment over a time range.

[0192] Training data may contain multiple training labels. Multiple training labels can identify multiple targets and / or interactions within a previously recorded training environment.

[0193] Figure 11 is a diagram of a labeling technique for generating target labels for a labeling environment 1100 according to some embodiments of the present disclosure. The training data may include data related to the labeling environment 1100 over a time range. First, the labeling environment 1100 may include at least one actor 1110 and a number of potential targets 1105A-D for the actor 1110. Additionally or alternatively, the training data may show trajectories 1115 recorded for actor 1110 over a time range. Actor 1110 may be an actual actor or a simulated actor (as shown in the recorded log data). Thus, the training data obtained may be log data of an actual actor or a simulation of an actor.

[0194] The labeling environment 1100 may be processed to generate multiple true and / or false labels for each of the potential targets 1105A-D based on the trajectory 1115 recorded for actor 1110. For this purpose, the labeling computing system may determine the average cross-sectional distance from the recorded trajectory 1115 to each potential target 1105A-D. For example, the average cross-sectional distance may include the average distance from each individual waypoint of the recorded trajectory 1115 to each waypoint of the target path of each potential target.

[0195] For example, a labeling computing system may determine multiple waypoints for a recorded trajectory 1115. Each of these waypoints may represent the individual positions of actor 1110 at individual times. The labeling computing system may determine one or more first distances between the first waypoint of the recorded trajectory 1115 and the individual first waypoints for each potential target 1105A-D. The first waypoint may correspond to the first time. The labeling computing system may determine different distances between the next waypoint of the recorded trajectory 1115 and the next individual waypoints for each potential target 1105A-D. For example, the next waypoint may correspond to the second time following the first time. This process can be repeated for each of the multiple waypoints of the recorded trajectory 1115. The average cross-sectional distance between the recorded trajectory 1115 and the individual potential targets may include the average distance between each waypoint of the recorded trajectory 1115 and the corresponding waypoint of each individual potential target.

[0196] The labeling computing system can compare the average cross-sectional distance between the recorded trajectory 1115 and each potential target 1105A-D with a threshold distance. If the average cross-sectional distance between the recorded trajectory 1115 and a particular target 1105A is less than the threshold distance, the labeling computing system may label that particular target 1105A as the True Goal. If the average cross-sectional distance between the recorded trajectory 1115 and other targets 1105B-D is greater than the threshold distance, the labeling computing system may label the other targets 1105B-D as False Goals. A particular target 1105A labeled as the True Goal can identify a potential target in the environment 1100 that actor 1110 is trying to move to.

[0197] The critical distance can be any distance. In some embodiments, the threshold distance can be determined based on the labeling environment. For example, the threshold distance may be based on map data corresponding to the labeling environment. For example, the threshold distance may be half the width of the individual lanes corresponding to potential targets 1105A-D.

[0198] In some embodiments, if the recorded trajectory 1115 of actor 1110 is close to the target path of a potential target for at least 50% of the observed future (e.g., the time range covered by the training sequence), the potential target may be labeled as true. For example, in the labeling environment 1100, actor 1110 can turn right according to the target path of a particular target 1105A. As a result, the particular target 1105A for the right turn may be labeled as true, and other potential targets may be labeled as false.

[0199] The data may include data related to the erased environment 110 and can be used to train at least a portion of the machine learning predictive model 706. For example, labeled targets (e.g., targets labeled as true or false) can be used to train a portion of the machine learning predictive model 706 (e.g., the decoder portion 730) that generates target probabilities. The machine learning predictive model 706 can be trained using training data based on data related to the labeled environment, for example, as illustrated with reference to Figures 15 and 16.

[0200] Labeled training data can be generated for training a machine learning predictive model 706 to predict interaction types. Figure 12 is an exemplary diagram of a potential interaction 700 according to some embodiments of the present disclosure. The potential interaction 700 may be based on a plurality of first potential targets 505A-D for a first actor 510A and a plurality of second potential targets 515A-D for a second actor 510B. The potential interaction 700 can define the temporal aspects of the intentions of the individual actors. For example, the potential interaction 700 between actors 510A-B can describe how each actor behaves in relation to the others. In this example, the first actor and the second actors 510A-B may be actual actors (as shown in the recorded log data) or simulated actors.

[0201] The interaction 700 can be identified based on the shared space 710 between the first actor 510A and the second actor 510B. The shared space 710 may include the region of space that both the first actor 510A and the second actor 510B intend to occupy according to their respective potential objectives. For example, the shared space 710 may be occupied by the first actor 510A if the first actor 510A decides to turn left according to a potential objective 505D. The shared space may also be occupied by the second actor 510B if the second actor 510B decides to go straight according to a potential objective 515B.

[0202] Training data can be labeled as discrete interaction types for potential interactions 700. Discrete interaction types may include concession or reverse concession interaction types, depending on which actor is predicted to reach the shared space 710 first. For example, the actor that first reaches the shared space 710 may be labeled as a reverse concession interaction type, and the actor that second reaches the shared space 710 may be labeled as a concession interaction type. The timing of each actor's arrival at the shared space 710 may be based on state data of the first actor 510A and the second actor 510B (e.g., speed, location, etc.) and / or map data of the environment 500 (e.g., right of way, traffic signal status, etc.). A third interaction category may include a ignore category that can be specified when there is no shared space between two potential targets (e.g., targets 505A and 515D) of the first actor 510A and the second actor 510B.

[0203] The labeled training data in Figure 12 can be used to train a machine learning predictive model (for example, following the processes in Figures 15 and 16). This allows the predictive system to leverage discrete interaction categories (e.g., reverse concession, concession, ignore) to describe the interactions between the first and second actors 510A and B and their potential goals.

[0204] Figure 13 is a flowchart of Method 1200 according to some embodiments of the present disclosure for initiating the movement of an autonomous vehicle. Method 1200 may be performed by a computing system including one or more computing devices, such as a computing system described with reference to other figures (e.g., an autonomous platform, an autonomous system, a predictive system, etc.). Each individual part of Method 1200 may be performed by any (or any combination) of the one or more computing devices. Also, one or more parts of Method 1200 may be implemented, for example, as an algorithm for the hardware components of the devices described herein (e.g., Figures 1-2, 4, 7, 18, etc.) for performing actor intent prediction as described herein. Figure 13 shows the elements to be performed in a particular order for illustrative and discussional purposes. Those skilled in the art will understand that using the disclosure provided herein, elements of any method described herein can be adjusted, rearranged, extended, omitted, combined or modified in various ways without departing from the scope of the present disclosure. Figure 13 is described with reference to elements / terms described in relation to other systems and drawings for illustrative purposes and is not intended to be limiting. One or more parts of Method 1200 may be carried out additionally or alternatively by other systems.

[0205] In step 1205, method 1200 includes the step of obtaining state data relating to multiple actors in the environment of an autonomous vehicle, and map data representing multiple lanes in the environment.

[0206] State data may include multiple actor states associated with an individual actor. These multiple actor states may indicate one or more positions, one or more velocities, and / or one or more directions of travel for each individual actor at the current time and one or more past time points.

[0207] In step 1210, method 1200 includes determining a plurality of potential goals, each of which has at least one potential goal, based on state data and map data. The potential goals may include the location of a potential destination in the environment and the target path to the location of the potential destination. For example, the potential goals may include a future location defined by the map data.

[0208] In some embodiments, the target path of a potential objective may include a nominal path defined in map data. For example, the map data may include multiple map features related to the potential objective. The multiple map features may include multiple waypoints for the nominal path defined in the map data.

[0209] In step 1215, method 1200 includes the step of determining (i) predicted targets for each actor of a plurality of actors (e.g., expressed in target probabilities), (ii) predicted interactions between each actor of the plurality of actors and other actors (e.g., expressed in interaction probabilities) based on the predicted targets, and (iii) continuous trajectories for each actor based on the predicted targets.

[0210] In some embodiments, a machine learning prediction model may determine the probability of each potential target for individual actors. The predicted target may be the individual potential target with the highest probability.

[0211] The predicted interactions may be based on the predicted goals for individual actors in the environment and the potential goals for other actors that intersect with them. The predicted interactions represent the probabilities of the predicted interactions between individual actors and other actors. For example, the predicted interactions may be a probability distribution for one or more interaction types (e.g., concession, reverse concession, neglect, ignorance). The probability distribution may be based on one or more distances between individual actors and the shared space related to the predicted goals and the potential goals of other actors.

[0212] In step 1220, method 1200 includes the step of initiating the movement of an autonomous vehicle based on predicted targets, predicted interactions, and / or sequential trajectories for individual actors. For example, a computing system may initiate the movement of an autonomous vehicle based on predicted targets, predicted interactions, and / or sequential trajectories for individual actors.

[0213] Figure 14 is a flowchart of Method 1300 according to some embodiments of the present disclosure for processing data with a machine learning predictive model. Method 1300 may be performed by a computing system including one or more computing devices, such as a computing system described with reference to other drawings (e.g., an autonomous platform, an autonomous system, a predictive system, etc.). Each individual part of Method 1300 may be performed by any device (or any combination) of the one or more computing devices. Also, one or more parts of Method 1300 may be implemented, for example, as an algorithm of the hardware components of the devices described herein (e.g., Figures 1-2, 4, 7, 18, etc.) for processing information with the machine learning intent model described herein. Figure 14 shows the elements to be performed in a particular order for illustrative and discussional purposes. Those skilled in the art will understand that using the disclosure provided herein, elements of any method described herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 14 is described with reference to elements / terms described in relation to other systems and drawings for illustrative purposes and is not intended to be limiting. One or more parts of Method 1300 may be carried out additionally or alternatively by other systems.

[0214] Method 1300 may include a sub-operation of operation 1215 in Figure 13, where Method 1200 includes the step of determining predicted targets, predicted interactions, and continuous trajectories by processing state data, map data, and multiple potential targets using a machine learning predictive model.

[0215] In step 1305, method 1300 includes processing state data to generate actor feature embeddings based on multiple actor states related to a first actor among multiple actors.

[0216] In step 1310, method 1300 includes processing map data to generate a target feature embedding based on a first target among a plurality of targets. The target feature embedding can encode a plurality of map features associated with the first potential target. For example, the plurality of map features may include map features (e.g., location, right of way, etc.) corresponding to each of a plurality of waypoints of the target route to the potential target.

[0217] In step 1315, method 1300 includes processing state data and map data to generate actor-target feature embeddings based on multiple actor states for a first target route.

[0218] In step 1320, method 1300 includes processing state data and map data to generate a target-target feature embedding based on multiple actor states for a second actor among multiple actors.

[0219] In step 1325, method 1300 includes the step of performing one or more message forwarding rounds to generate multiple updated feature embeddings.

[0220] As an example, each feature embedding may be associated with a node and / or edge of a graph neural network. For instance, a machine learning prediction model may include a graph neural network containing multiple nodes and multiple edges. Multiple nodes may include (i) multiple actor nodes, each corresponding individually to a different actor of multiple actors, and (ii) multiple target nodes, each corresponding individually to a different potential goal of multiple potential goals. Multiple edges may include (i) one or more actor-target edges, each individually connecting an individual actor node to an individual target node, and (ii) one or more target-target edges, each individually connecting at least two of the multiple target nodes. During each message transfer round, each node and edge of the graph neural network may share information with neighboring nodes and / or edges. As described herein, a machine learning prediction model may leverage a learned model / function (e.g., a target-target edge model, an actor-target edge model, a target node model) to perform message transfers that allow feature information to be shared between nodes / edges.

[0221] In step 1330, method 1300 includes the step of processing at least one of actor feature embeddings or actor-target feature embeddings to determine the predicted target for the first actor.

[0222] In step 1335, method 1300 includes the step of processing goal-goal feature embeddings to determine the interaction between the first actor and other actors based on the predicted goal.

[0223] Additionally or alternatively, multiple feature embeddings can be handled in a conditional machine learning intent prediction system according to exemplary method 1400.

[0224] Figure 15 is a flowchart of another exemplary method 1400 according to some embodiments of the present disclosure for processing data with a machine learning predictive model. Method 1400 may be carried out by a computing system including one or more computing devices, such as a computing system described with reference to other drawings (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual part of Method 1400 may be carried out by any device (or any combination) of the one or more computing devices. Also, one or more parts of Method 1400 may be implemented, for example, as an algorithm of the hardware components of the devices described herein (e.g., Figures 1-2, 4, 7, 18, etc.) for processing information with the machine learning intent model described herein. Figure 15 shows the elements to be performed in a particular order for illustrative and discussion purposes. Those skilled in the art will understand that using the disclosure provided herein, elements of any method described herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 15 is described with reference to elements / terms described in relation to other systems and drawings for illustrative purposes and is not intended to be limiting. One or more parts of Method 1400 may be carried out additionally or alternatively by other systems.

[0225] Method 1400 may include a sub-operation of operation 1215 in Figure 13, where Method 1200 includes the step of determining predicted targets, predicted interactions, and sequential trajectories by processing state data, map data, and multiple potential targets using a machine learning predictive model. For example, Method 1400 may include operations that may occur concurrently with operations 1305-1325 in Figure 14, where Method 1300 includes the step of performing one or more message forwarding rounds to generate multiple updated feature embeddings. Method 1400 may include operations that may occur before operation 1220 in Figure 13, where Method 1200 includes the step of initiating the movement of an autonomous vehicle based on predicted targets, predicted interactions, and / or sequential trajectories for individual actors.

[0226] In step 1405, method 1400 includes the step of receiving vehicle motion data for an autonomous vehicle. The vehicle motion data may indicate the potential movements of the autonomous vehicle. For example, the vehicle motion data may indicate a potential objective selected for implementation by the autonomous vehicle.

[0227] In step 1410, method 1400 includes the step of generating a conditional feature embedding based on the intended movement of the autonomous vehicle.

[0228] In step 1415, method 1400 includes the step of concatenating a conditional feature embedding to at least one of a plurality of feature embeddings.

[0229] In step 1420, method 1400 includes the step of determining predicted targets, predicted interactions, and continuous trajectories for individual actors by processing multiple feature embeddings using a machine learning predictive model. For example, a computing system may determine predicted targets, predicted interactions, and continuous trajectories for individual actors by processing multiple feature embeddings with a machine learning conditional predictive model.

[0230] Figure 16 is a flowchart of an exemplary method 1500 according to some embodiments of the present disclosure for training a machine learning predictive model. Method 1500 may be performed by a computing system including one or more computing devices, such as a computing system described with reference to other figures (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual part of Method 1500 may be performed by any device (or any combination) of the one or more computing devices. Also, one or more parts of Method 1500 may be implemented, for example, as an algorithm of the hardware components of the devices described herein (e.g., Figures 1-2, 4, 7, 18, etc.) for training the machine learning intent model described herein. Figure 16 shows the elements to be performed in a particular order for illustrative and discussion purposes. Those skilled in the art will understand that using the disclosure provided herein, elements of any method described herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 16 is described for illustrative purposes with reference to elements / terms described in relation to other systems and drawings and is not intended to be limiting. One or more parts of Method 1500 may be carried out additionally or alternatively by other systems.

[0231] In step 1505, method 1500 includes the step of generating or acquiring training data. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) may generate the training data, or the training data may be acquired from a real-world log. The training data may include labeled training data, such as data related to the labeling environment described herein. The labeled training data can identify multiple actors, multiple potential goals for each actor, and whether each actor has decided to act according to each individual goal. In some embodiments, the training data may include a previously recorded training environment.

[0232] In step 1510, method 1500 includes the step of inputting at least a portion of the training data into a machine learning predictive model in order to determine a training predictive goal. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) can input at least a portion of the training data into a machine learning predictive model in order to determine a training predictive goal.

[0233] In step 1515, method 1500 includes the step of determining the loss of a machine learning predictive model based on a training predictive goal and a set of target labels. For example, a computing system (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.) may determine the loss for a machine learning predictive model based on a training predictive goal and a set of target labels.

[0234] In step 1520, method 1500 includes the step of updating one or more parameters of a machine learning predictive model based on the loss. For example, a computing system can update one or more parameters of a machine learning predictive model based on the loss.

[0235] Figure 17 is a flowchart of an exemplary method 1600 according to some embodiments of the present disclosure for generating training data. Method 1600 may be performed by a computing system including one or more computing devices, such as a computing system described with reference to other drawings (e.g., an autonomous platform, an autonomous system, an intent prediction system, etc.). Each individual part of Method 1600 may be performed by any device (or any combination) of the one or more computing devices. Also, one or more parts of Method 1600 may be implemented, for example, as an algorithm of the hardware components of the devices described herein (e.g., Figures 1-2, 4, 7, 18, etc.) for generating the training data described herein. Figure 17 shows the elements to be performed in a particular order for illustrative and discussion purposes. Those skilled in the art will understand that using the disclosure provided herein, elements of any method described herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 17 is described with reference to elements / terms described in relation to other systems and drawings for illustrative purposes and is not intended to be limiting. One or more parts of Method 1600 may be carried out additionally or alternatively by other systems.

[0236] In 1605, method 1600 includes the step of acquiring training scene data about the training environment in a first time step. For example, the training scene data may include real-world data or simulated data recorded by one or more sensors of the autonomous platform.

[0237] In step 1610, method 1600 includes the step of generating multiple actor labels and goal labels for the training environment. The multiple goal labels identify multiple goals for the training actors.

[0238] In step 1615, method 1600 includes the step of acquiring training scene data about the training environment over a time range that identifies the actual trajectory of the training actor.

[0239] In step 1620, method 1600 includes the step of calculating the mean-track distance between the actual trajectory and the individual target paths for each of the actor's multiple training objectives.

[0240] In step 1625, method 1600 includes the step of updating multiple actor labels and target labels for the training environment.

[0241] Figure 18 is a block diagram of an exemplary computing ecosystem 10 according to an exemplary embodiment of the present disclosure. The exemplary computing ecosystem 10 may include a first computing system 20 and a second computing system 40 that are communicably coupled over one or more networks 60. In some embodiments, the first computing system 20 or the second computing system 40 may implement one or more of the systems, operations, or functions described herein for intent prediction (e.g., a remote system 160, an onboard computing system 180, an autonomous system 200, etc.).

[0242] In some embodiments, the first computing system 20 may be included in an autonomous platform and utilized to perform functions of the autonomous platform as described herein. For example, the first computing system 20 may be onboard a self-driving vehicle and can implement an autonomous system for autonomously operating the self-driving vehicle. In some embodiments, the first computing system 20 may represent an entire onboard computing system or a part thereof (e.g., a position estimation system 230, a perception system 240, a planning system 250, a control system 260, or a combination thereof). In other embodiments, the first computing system 20 may not be onboard a self-driving platform. The first computing system 20 may include one or more independent physical computing devices 21.

[0243] The first computing system 20 (for example, its computing device 21) may include one or more processors 22 and memory 23. The one or more processors 22 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operationally connected processors. The memory 23 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

[0244] Memory 23 may store information accessible by one or more processors 22. For example, memory (e.g., one or more non-temporary computer-readable storage media, memory devices, etc.) may store data 24 that can be retrieved (e.g., received, accessed, written, manipulated, formed, generated, stored, pulled, downloaded, etc.). Data 24 may include, for example, sensor data, map data, data related to autonomous functions (e.g., data related to recognition, planning, or control functions), simulation data, or any data or information described herein. In some embodiments, the first computing system 20 may retrieve data from one or more memory devices remotely from the first computing system 20.

[0245] Memory 23 may store computer-readable instructions 25 that can be executed by one or more processors 22. Instructions 25 can be implemented by software or hardware written in any suitable programming language. Additionally or alternatively, instructions 25 may be executed by logically or virtually isolated threads on the processors 22.

[0246] For example, memory 23 may store instructions 25 that can be executed by one or more processors (e.g., one or more processors 22, one or more other processors) to perform any of the operations, functions, or methods / processes (or parts thereof) described herein (e.g., in a computing device 21, a first computing system 20, or another system having a processor that executes instructions).

[0247] In some embodiments, the first computing system 20 may store or include one or more models 26. In some embodiments, the model 26 may be one or more machine learning models, or otherwise include them. For example, the model 26 may be a variety of machine learning models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear and nonlinear models, or otherwise include them. Examples of neural networks include forward neural networks, cyclic neural networks (e.g., long-short-term memory cyclic neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system 20 may include one or more models for implementing subsystems of an autonomous system 200, which may include a location estimation system 230, a recognition system 240, a planning system 250, or a control system 260.

[0248] In some embodiments, the first computing system 20 may acquire one or more models 26 that communicate with the second computing system 40 via a network 60 using a communication interface 27. For example, the first computing system 20 may store the models (e.g., one or more machine learning models) in memory 23. The first computing system 20 can then use or otherwise implement the models 26 (e.g., by a processor 22). For example, the first computing system 20 may implement the models 26 to estimate the position of an autonomous platform in an environment, recognize the environment of the autonomous platform or objects within it, plan one or more future states of the autonomous platform for movement through the environment, and control the autonomous platform to interact with the environment.

[0249] The second computing system 40 may include one or more computing devices 41. The second computing system 40 may include one or more processors 42 and memory 43. The one or more processors 42 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operationally connected processors. The memory 43 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

[0250] Memory 43 may store information accessible by one or more processors 42. For example, memory 43 (e.g., one or more non-temporary computer-readable storage media, memory devices, etc.) may store retrieval data 44. The data 44 may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data related to vehicle trips / services, or any data or information described herein. In some embodiments, a second computing system 40 may retrieve data from one or more memory devices remotely from the second computing system 40.

[0251] Memory 43 may also store computer-readable instructions 45 that can be executed by one or more processors 42. Instructions 45 can be implemented by software or hardware written in any suitable programming language. Additionally or alternatively, instructions 45 may be performed in logically or virtually isolated threads on the processors 42.

[0252] For example, memory 43 may store instructions 45 that can be executed (e.g., by one or more processors 42, one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., by computing device 41 such as computing device 21 or first computing system 20, second computing system 40, or other system having a processor for executing instructions). This may include, for example, functions of the autonomous system 200 (e.g., position estimation, recognition, planning, control, etc.) or other functions related to the autonomous platform (e.g., remote assistance, mapping, vehicle management, trip / service assignment and matching, etc.).

[0253] In some embodiments, the second computing system 40 may include one or more server computing devices. If the second computing system 40 includes multiple server computing devices, these server computing devices can operate according to various computing architectures, including, for example, sequential computing architectures, parallel computing architectures, or combinations thereof.

[0254] In addition to or as an alternative to Model 26 of the first computing system 20, the second computing system 40 may include one or more Models 46. For example, Model 46 may be one or more machine learning models, or otherwise may include them. As an example, Model 26 may be a variety of machine learning models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear and nonlinear models, or otherwise may include them. Examples of neural networks include forward neural networks, cyclic neural networks (e.g., long-short-term memory cyclic neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system 40 may include one or more models of the autonomous system 200.

[0255] In some embodiments, a second computing system 40 or a first computing system 20 can train one or more machine learning models of model 26 or model 46 using one or more model trainers 47 and training data 48. A model trainer 47 can train either model 26 or model 46 using one or more training or learning algorithms. One example of a training technique is error backpropagation. In some embodiments, a model trainer 47 may perform a guided training technique using labeled training data. In other embodiments, a model trainer 47 may perform an unguided training technique using unlabeled training data. In some embodiments, the training data 48 may include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some embodiments, a second computing system 40 can perform a simulation to obtain the training data 48 or to run a model trainer 47 to train or test model 26 or model 46. For example, the model trainer 47 can train one or more components of a machine learning model for the autonomous system 200 through unsupervised training techniques that use a target function (e.g., cost, compensation, heuristics, constraints, etc.). In some embodiments, the model trainer 47 may perform multiple generalization techniques to improve the generalization ability of the model under training. Generalization techniques include weight decay, dropouts, or other techniques.

[0256] The first computing system 20 and the second computing system 40 may each include communication interfaces 27 and 49, respectively. The communication interfaces 27 and 49 can be used to communicate with each other or with one or more other systems or devices, including systems or devices located remotely from the first computing system 20 or the second computing system 40. The communication interfaces 27 and 49 may include any circuits, components, software, etc., for communicating with one or more networks (e.g., network 60). In some embodiments, the communication interfaces 27 and 49 may include, for example, one or more communication controllers, receivers, transceivers, transmitters, ports, conductors, software, or hardware for data communication.

[0257] Network 60 may be any type of network or combination of networks that enables communication between devices. In some embodiments, the network may include one or more of the following: a local area network, a wide area network, the Internet, a secure network, a cellular network, a mesh network, a peer-to-peer communication link, or some combination thereof, and may include any number of wired or wireless links. Communication over Network 60 can be achieved, for example, through a network interface using any type of protocol, protection scheme, encoding, format, packaging, etc.

[0258] Figure 18 shows an example computing ecosystem 10 that can be used to implement this disclosure. Other systems can also be used. For example, in some embodiments, the first computing system 20 may include a model trainer 47 and training data 48. In such embodiments, models 26, 46 can all be trained and used locally on the first computing system 20. As another example, in some embodiments, the computing system 20 may not be connected to any other computing system. Also, components that are exemplified or discussed as being included in one of the computing systems 20 or 40 may instead be included in the other one of the computing systems 20 or 40.

[0259] Computing tasks discussed herein as being performed on a remote computing device from an autonomous platform (e.g., an autonomous vehicle) may, conversely, be performed on the autonomous platform itself (e.g., via the vehicle computing system of the autonomous vehicle), and vice versa. Such configurations can be implemented without departing from the scope of this disclosure. Computer-based systems allow for a wide variety of possible configurations, combinations, and divisions between components. Operations performed by a computer may be performed on a single component or across several components. Tasks or operations performed by a computer may be performed sequentially or in parallel. Data and instructions may be stored in a single memory device or across multiple memory devices.

[0260] The aspects of this disclosure are described in terms of their exemplary embodiments. A person skilled in the art will be able to devise numerous other embodiments, variations, or modifications within the appended claims and spirit by considering this disclosure. Any features described in the following claims can be combined or rearranged in all possible ways. Therefore, the scope of this disclosure is exemplary and not restrictive, and this disclosure does not exclude such variations, modifications, or additions to the subject matter which are obvious to a person skilled in the art. Terms are described herein using lists of exemplary elements linked by conjunctions such as “and,” “or,” and “but.” These conjunctions should be understood to be provided for illustrative purposes only. For example, a list linked by a particular conjunction such as “or” may mean “at least one” or “any combination” of the exemplary elements enumerated therein, and unless otherwise explicitly stated, “or” should be understood as “and / or.” Similarly, terms such as “based on” should be understood as “based at least partially.”

[0261] Those skilled in the art will understand that any element of the claims, actions, or processes discussed herein using the disclosures provided herein can be adjusted, rearranged, expanded, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Some claims are described with letter references to claim elements for illustrative purposes and not to limit them. Letter references do not imply a particular order of actions. For example, letter identifiers such as (a), (b), (c), ..., (i), (ii), (iii), ... can be used to describe actions. These identifiers are provided for the convenience of the reader and do not indicate a particular order of steps or actions. Actions exemplified by list identifiers such as (a), (i) may occur before, after, or in parallel with other actions exemplified by list identifiers such as (b), (ii).

Claims

1. A computer-implemented method, (a) A step of acquiring state data related to multiple actors in the environment of an autonomous vehicle and map data representing multiple lanes in the environment, (b) A step of determining a plurality of potential targets, including at least one potential target for each individual actor of the plurality of actors, based on the state data and the map data, the potential targets include the location of a potential destination in the environment and the target path to the location of the potential destination, (c) Processing the state data, the map data, and the plurality of potential targets using a machine learning prediction model to determine (i) predicted targets for each actor of the plurality of actors, (ii) predicted interactions between each actor of the plurality of actors and other actors based on the predicted targets, and (iii) a continuous trajectory for each actor based on the predicted targets - the machine learning prediction model includes a graph neural network including a plurality of nodes and a plurality of edges, wherein the plurality of nodes includes (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors, and (ii) a plurality of target nodes corresponding individually to each of the plurality of potential targets, and the plurality of edges includes (iii) one or more actor-target edges that individually connect individual actor nodes and individual target nodes, and (iv) one or more target-target edges that individually connect at least two of the plurality of target nodes - (d) A computer implementation method comprising the step of initiating the movement of an autonomous vehicle based on the predicted target, the predicted interaction, or the continuous trajectory for each of the actors.

2. (c) Step is, The computer implementation method according to claim 1, further comprising the step of determining the probability of each potential target for each individual actor, wherein the predicted targets include the individual potential targets having the highest probability.

3. The computer implementation method according to claim 1, wherein the target path of the potential target includes a nominal path defined in the map data.

4. The computer implementation method according to claim 3, wherein the map data includes a plurality of map features relating to the potential target, and the plurality of map features include a plurality of waypoints for the nominal route defined in the map data.

5. The computer implementation method according to claim 1, wherein the predicted interactions are further based on potential targets for other actors that intersect with the predicted targets for the individual actors in the environment.

6. The computer implementation method according to claim 5, wherein the predicted interaction includes a probability distribution for one or more interaction types between the individual actors and the other actors.

7. The computer implementation method according to claim 6, wherein the probability of the predicted interaction between the individual actors and the other actors is based on one or more distances between the individual actors and a shared space relating to the predicted target and the other actors' potential targets.

8. The computer implementation method according to claim 1, wherein the state data includes a plurality of actor states related to the individual actors, and the plurality of actor states indicate one or more positions, one or more velocities, or one or more directions of travel (headings) for the individual actors at the present time and one or more past time points.

9. (c) Step is, The first part of the machine learning prediction model processes the state data, the map data, and the multiple potential targets to generate multiple feature embeddings corresponding to the multiple actors and the multiple potential targets. The computer implementation method according to claim 1, comprising the step of processing the plurality of feature embeddings to determine the predicted targets for each of the actors, the predicted interactions between the actors and other actors based on the predicted targets, and the continuous trajectories of the actors based on the predicted targets.

10. The computer implementation method according to claim 9, wherein the plurality of feature embeddings include an actor feature embedding corresponding to an actor node, a target feature embedding corresponding to a target node, an actor-target feature embedding corresponding to an actor-target edge, and a target-target feature embedding corresponding to a target-target edge.

11. (c) Step is, The steps include: determining the predicted target for each actor by processing at least one of the actor feature embedding or the actor-target feature embedding; The computer implementation method according to claim 10, comprising the step of processing the objective-objective feature embedding to determine the predicted interactions between the individual actors and other actors based on the predicted objective.

12. The steps include receiving vehicle motion data for the autonomous vehicle (the vehicle motion data indicates the potential movement of the autonomous vehicle), The steps include generating conditional feature embeddings based on the potential movements of the autonomous vehicle, The steps include: linking the conditional feature embedding to at least one of the plurality of feature embeddings; The computer implementation method according to claim 9, further comprising the step of determining the predicted target, the predicted interaction, and the continuous trajectory for each of the actors by processing the plurality of feature embeddings using a machine learning conditional prediction model.

13. (c) Step is, The steps include performing one or more message forwarding rounds to generate multiple updated feature embeddings, The computer implementation method according to claim 9, comprising the step of determining the predicted targets, predicted interactions, and continuous trajectories for each actor based on the plurality of updated feature embeddings.

14. A computing system, One or more processors, The system includes one or more non-temporary computer-readable media that store instructions that can be executed by one or more processors to perform an operation, and the operation is (a) An operation to acquire state data related to multiple actors in the environment of an autonomous vehicle and map data representing multiple lanes in the environment, (b) An operation to determine a plurality of potential targets, including at least one potential target for each of the plurality of actors, based on the state data and the map data, the potential targets include the location of the potential destination in the environment and the target path to the location of the potential destination, (c) Processing the state data, the map data, and the plurality of potential targets using a machine learning prediction model to determine (i) the predicted targets for each actor of the plurality of actors, (ii) the predicted interactions between each actor of the plurality of actors and other actors based on the predicted targets, and (iii) the operation of determining a continuous trajectory for each actor based on the predicted targets - the machine learning prediction model includes a graph neural network including a plurality of nodes and a plurality of edges, wherein the plurality of nodes includes (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors, and (ii) a plurality of target nodes corresponding individually to each of the plurality of potential targets, and the plurality of edges includes (iii) one or more actor-target edges that individually connect individual actor nodes and individual target nodes, and (iv) one or more target-target edges that individually connect at least two of the plurality of target nodes - (d) A computing system including an action to initiate the movement of an autonomous vehicle based on the predicted target, the predicted interaction, or the sequential trajectory for each of the actors.

15. (c) The operation is, The computing system according to claim 14, further comprising the operation of determining the probability of each potential target for each individual actor, wherein the predicted targets include the individual potential targets having the highest probability.

16. The computing system according to claim 14, wherein at least one actor represents the autonomous vehicle, and the state data relates to at least one of the position estimation system or inertial measurement device of the autonomous vehicle.

17. The computing system according to claim 16, wherein the map data includes a plurality of map features relating to the potential target, and the plurality of map features include a plurality of waypoints for a nominal route defined in the map data.

18. A control system for an autonomous vehicle, wherein the control system is One or more processors, The system includes one or more non-temporary computer-readable media that store instructions that can be executed by one or more processors to perform an operation, and the operation is (a) An operation to acquire state data related to multiple actors in the environment of the autonomous vehicle and map data representing multiple lanes in the environment, (b) An operation to determine a plurality of potential targets, including at least one potential target for each of the plurality of actors, based on the state data and the map data, the potential targets include the location of the potential destination in the environment and the target path to the location of the potential destination, (c) Processing the state data, the map data, and the plurality of potential targets using a machine learning prediction model to determine (i) the predicted targets for each actor of the plurality of actors, (ii) the predicted interactions between each actor of the plurality of actors and other actors based on the predicted targets, and (iii) the operation of determining a continuous trajectory for each actor based on the predicted targets - the machine learning prediction model includes a graph neural network including a plurality of nodes and a plurality of edges, wherein the plurality of nodes includes (i) a plurality of actor nodes corresponding individually to each actor of the plurality of actors, and (ii) a plurality of target nodes corresponding individually to each of the plurality of potential targets, and the plurality of edges includes (iii) one or more actor-target edges that individually connect individual actor nodes and individual target nodes, and (iv) one or more target-target edges that individually connect at least two of the plurality of target nodes - (d) An autonomous vehicle control system comprising: an action to initiate the movement of the autonomous vehicle based on the predicted target, the predicted interaction, or the continuous trajectory of the individual actors.

19. (c) The operation is, The control system for an autonomous vehicle according to claim 18, further comprising the operation of determining the probability of each potential target for each of the individual actors, wherein the predicted targets include the individual potential targets having the highest probability.

Images