A method for generating candidate planned trajectories and predicted future trajectories using a shared predictive neural network.

Abstract

This provides a method for generating candidate planned trajectories and predicted future trajectories using a shared predictive neural network. [Solution] A method, system, and apparatus comprising a computer program encoded on a computer storage medium for planning the future trajectory of an autonomous vehicle in an environment.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 735,270, filed December 17, 2024. The disclosure of the prior application is considered a part of this application and is incorporated by reference into the disclosure of this application.

Background Art

[0002] This specification relates to planning future trajectories of autonomous vehicles in an environment.

[0003] The environment may be a real - world environment, and the autonomous vehicle can be, for example, a vehicle in the environment.

[0004] Autonomous vehicles include fully autonomous cars, boats, and aircraft. Autonomous vehicles use various on - board sensors and computer systems to detect nearby objects and use such detections to make control and navigation decisions.

Summary of the Invention

[0005] This specification describes a method for an autonomous vehicle, e.g., a car, to plan a trajectory for the autonomous vehicle from perception signals, i.e., from the output generated by the on - board perception system of the autonomous vehicle, using a trained machine - learning model. More specifically, this specification describes using the same shared prediction neural network to generate both a candidate planned trajectory output for the autonomous vehicle and a predicted future trajectory output for other agents.

[0006] In general, one innovative aspect of the subject matter described herein may be embodied in a method comprising: the autonomous vehicle taking the following actions: acquiring scene data characterizing the scene in the environment at the present moment, including the autonomous vehicle and a set of target agents; the autonomous vehicle receiving data characterizing the intended path through the autonomous vehicle's environment after the present moment; and the autonomous vehicle processing the path data and scene data using a predictive neural network to generate predictive outputs, each including (i) a set of one or more candidate planned trajectory outputs for the autonomous vehicle, each specifying a planned trajectory for the autonomous vehicle, starting from the present moment; and (ii) a set of one or more predicted future trajectory outputs, each specifying a predicted future trajectory for each target agent in the set of target agents, starting from the present moment.

[0007] The embodiments described above, and other embodiments, may each optionally include one or more of the following features, either individually or in combination. In particular, one embodiment may include all of the following features in combination.

[0008] In some implementations, the method further includes using the predictive output to control the autonomous vehicle.

[0009] In some implementations, the current scene includes multiple other agents, and the set of target agents is a subset of those other agents.

[0010] In some implementations, the subset is a suitable subset.

[0011] In some implementations, each candidate planned trajectory output includes data that defines the planned future state of the autonomous vehicle at each of several future time steps.

[0012] In some implementations, each candidate plan trajectory output further includes a probability score representing the likelihood that each candidate plan trajectory is the optimal trajectory for the autonomous vehicle, given the intended path for the autonomous vehicle.

[0013] In some implementations, for each target agent, each predicted future trajectory output includes data defining the predicted future state of each target agent at each of several future time steps.

[0014] In some implementations, each predicted future trajectory output further includes a probability score, which represents the predicted probability that each predicted future trajectory is the actual trajectory traversed by the target agent.

[0015] In some implementations, a predictive neural network includes an encoder neural network that processes scene data to produce an encoded representation of the scene data, and a decoder neural network that processes the encoded representations of the scene data and path data to produce a predictive output.

[0016] In some implementations, each set of predicted trajectory outputs is adjusted for the encoded representation of the scene data but not for the path data, while the set of planned trajectory outputs is adjusted for both the encoded representation of the scene data and the path data.

[0017] In some implementations, the decoder neural network is configured to maintain a corresponding query for each candidate plan trajectory output and each predicted trajectory output, and to process the encoded representations of the scene data and path data to generate the predicted output, which includes updating the query corresponding to the candidate planner trajectory, adjusted for each candidate plan trajectory, for any other candidate plan trajectory, predicted trajectory, encoded representation of the scene data, and intended path, and generating candidate plan trajectories from the updated query.

[0018] In some implementations, processing the encoded representation of scene data and path data to generate a prediction output includes updating the query corresponding to the predicted trajectory for each predicted future trajectory, and updating the query corresponding to the predicted future trajectory, which is adjusted based on the query corresponding to the encoded representation of the scene data, but not on the intended path or any other candidate plan trajectory, and generating the predicted future trajectory from the updated query.

[0019] In some implementations, a decoder neural network processes an encoded representation of scene data and path data to generate a prediction output, and includes a shared prediction head that generates each candidate planned trajectory output and each predicted future trajectory output.

[0020] In some implementations, each candidate planned trajectory output identifies the respective directional signal state for the autonomous vehicle at each future time step of each planned trajectory, given that the autonomous vehicle is following each planned trajectory.

[0021] In some implementations, each candidate planned trajectory output identifies the respective gear state for the autonomous vehicle at each future time step of each planned trajectory, given that the autonomous vehicle is following each planned trajectory.

[0022] In some implementations, the prediction output further includes, for each of the one or more traffic signals in the scene, the predicted state of each traffic signal at each of the one or more future points in time after the present.

[0023] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the acts of the method. The fact that one or more computer systems are configured to perform a particular operation or act means that during operation, the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operation or act. The fact that one or more computer programs are configured to perform a particular operation or act means that when executed by a data processing device, the one or more programs include instructions that cause the device to perform the operation or act.

[0024] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is a block diagram of an exemplary system. [Figure 2] FIG. 2 is a flowchart of an exemplary process for controlling an autonomous vehicle. [Figure 3] FIG. 3 is a diagram showing the architecture of a predictive neural network. [Figure 4] FIG. 4 is a flowchart of an exemplary process for generating a candidate planned trajectory and a predicted future trajectory.

Modes for Carrying Out the Invention

[0026] In the various drawings, like reference numbers and designations indicate like elements.

[0027] This specification describes a method by which an autonomous vehicle, such as an automobile, plans its trajectory from perceptual signals, i.e., outputs generated by the autonomous vehicle's onboard perceptual system, using a trained machine learning model.

[0028] Autonomous vehicles generally perform both behavior prediction, i.e., predicting the behavior of other agents in their vicinity, and trajectory planning, i.e., planning the vehicle's trajectory to effectively navigate the driving environment. These tasks are interrelated in that the future behavior of other agents influences the optimal future trajectory of the autonomous vehicle, and the future behavior of the autonomous vehicle influences the future behavior of other agents. For example, the optimal future trajectory of an autonomous vehicle may include deceleration if another agent merges into the autonomous vehicle's driving line ahead of it.

[0029] However, existing systems generally perform these tasks using separate machine learning models. That is, an autonomous driving system may use one neural network to perform behavior prediction and another neural network to perform trajectory planning. For example, this may be done because it is difficult to train a single neural network to perform both tasks simultaneously. In a specific example, this may be done because trajectory planning is tailored based on privileged information unknown to other agents, such as the intended path of the autonomous vehicle, while behavior prediction needs to be performed independently of this privileged information in order to accurately model the future behavior of other agents, i.e., because other agents do not actually know the intended path of the autonomous vehicle in the future. However, performing these tasks separately results in suboptimal trajectory planning results and introduces additional delays and computer overhead into the prediction process, i.e., because the trajectory planning output cannot take into account information from the behavior prediction output, and because it is necessary to maintain and perform inference using two separate neural networks.

[0030] The techniques described herein address these issues by using a shared predictive neural network that generates both candidate planned trajectories and predicted future trajectories in parallel from scene-characterizing data. This requires only one neural network to perform both tasks, reduces latency, improves the quality of the candidate planned trajectories because the candidate planned trajectories effectively incorporate context from the predicted future trajectories. Furthermore, the techniques described can perform this parallel prediction while maintaining the accuracy of the predicted future trajectories by avoiding the adjustment of the predicted future trajectories on privileged information. Instead, the techniques described implement a dependency scheme that enables the predictive neural network to operate so that (i) each set of predicted trajectory outputs is adjusted on the current scene in the environment but not on the intended path of the autonomous vehicle, while (ii) the set of planned trajectory outputs is adjusted on the current scene and the intended path. Thus, the resulting planned trajectory outputs and the resulting predicted trajectory outputs can be generated with reduced computer overhead, improved accuracy, and improved control of the autonomous vehicle.

[0031] Furthermore, in some cases, the described techniques enable a shared predictive neural network to generate one or more additional outputs for each candidate planned trajectory. Generating these additional outputs can provide a richer training signal for training the predictive neural network, and can provide additional information that enables a control system to use the predictions generated by the predictive neural network more effectively to control the autonomous vehicle, or both.

[0032] Figure 1 shows an exemplary system 100. System 100 is an embodiment of a system implemented as a computer program on one or more computers in one or more locations, which implements the systems, components, and technologies described later. System 100 includes an onboard system 110 and a training system 120.

[0033] The onboard system 120 is physically positioned onboard the vehicle 102. Being onboard the vehicle 102 means that the onboard system 110 includes components that move with the vehicle 102, such as power supplies, computing hardware, and sensors. In some cases, the vehicle 102 is an autonomous vehicle. An autonomous vehicle can be a fully autonomous vehicle that makes and executes fully autonomous driving decisions to navigate within its environment. An autonomous vehicle can also be a semi-autonomous vehicle that uses predictions to assist a human driver. For example, if a prediction indicates that a human driver is about to collide with another vehicle, the vehicle 102 may autonomously apply the brakes. In another example, the vehicle 102 may have an advanced driver-assistance system (ADAS) that assists the human driver of the vehicle 102 in driving the vehicle 102 by detecting potentially dangerous situations and warning the human driver or otherwise responding to the dangerous situation. As a specific example, vehicle 102 may warn its driver or perform autonomous driving actions when an obstacle is detected, when the vehicle deviates from its driving lane, or when an object is detected in the blind spot of the human driver.

[0034] The onboard system 110 includes one or more sensor systems 130. The sensor systems 130 may include a combination of components that receive reflections of electromagnetic radiation, such as a lidar system that detects reflections of laser light, a radar system that detects reflections of radio waves, and a camera system that detects reflections of visible light.

[0035] Sensor data generated by a given sensor generally indicates the distance, direction, and intensity of reflected radiation. For example, a sensor can transmit one or more pulses of electromagnetic radiation in a specific direction and measure the intensity of any reflection and the time it takes for the reflection to be received. The distance can be calculated by determining the time elapsed between the pulse and its corresponding reflection. Each sensor can continuously sweep a particular space in terms of angle, azimuth, or both. Sweeping in terms of azimuth can, for example, allow a sensor to detect multiple objects along the same line of sight.

[0036] The sensor system 130, or other components of the vehicle 102, may also classify a group of one or more raw sensor measurements from one or more sensors as measurements from another agent. The group of sensor measurements may be represented in one of several ways, depending on the type of sensor measurement being captured. For example, each group of raw laser sensor measurements may be represented as a three-dimensional point cloud, where each point may have intensity and position in a specific two-dimensional or three-dimensional coordinate space. In some implementations, the position is represented as a pair of range and elevation. Each group of camera sensor measurements may be represented as an image patch, for example, an RGB image patch.

[0037] When the sensor system 130 classifies one or more groups of raw sensor measurements as measures for each of the other agents, the sensor system 130 can compile the raw sensor measurements into a set of raw data 132 and transmit the raw data 132 to the data representation system 140.

[0038] Furthermore, the data representation system 140 mounted on the vehicle 102 receives raw sensor data 132 from the sensor system 130 and generates scene data 142. The scene data 142 characterizes the current state of the environment surrounding the vehicle 102 at the present time.

[0039] For example, scene data may characterize the current state at the present moment and the previous state at one or more previous points in time for one or more agents in the environment, such as other vehicles, pedestrians, or cyclists. In other words, scene data may include data characterizing the previous trajectories of one or more agents in the environment up to the present moment. The state of an agent at a given point in time, such as a vehicle, pedestrian, or cyclist, may include the agent's position at that point in time and, optionally, values ​​for a predetermined set of behavioral parameters at that point in time. In a specific example, behavioral parameters may include the agent's orientation, velocity, and / or acceleration. In another embodiment, one or more agents in the environment may be represented together by a state that describes a combined representation of, for example, their position, dimensions, and velocity (e.g., the position, velocity, and horizontal dimensions of a group of pedestrians).

[0040] In some implementations, the scene data may also include data characterizing the current state of the autonomous vehicle at the present moment, and the previous state of the autonomous vehicle at one or more previous points in time.

[0041] In some implementations, scene data also includes data that characterizes the features of the environment. These features may include (i) dynamic features of the environment at the present moment, such as the state of traffic signals; (ii) static features of the environment, such as road graph data that characterizes one or more of the roads near the autonomous vehicle, such as lane connections, lane types, stop lines, and speed limits; or (iii) both. For example, dynamic features of the environment may be represented by a combination of polylines (e.g., feature locations) and one-hot encoding vectors (e.g., one-hot encoding of red-yellow-green traffic signals, polylines representing locations in the environment). Static features of the environment may be represented by polylines. For features represented by polylines, the features may be converted into feature vectors (e.g., represented by x, y points with vector directions in the environment, such as x, y positions and numerical values ​​representing orientation) before the system uses them.

[0042] The data representation system 140 provides the scene data 142 to the track planning system 200, which is also installed on the vehicle 102.

[0043] Furthermore, the route planning system 144 provides route data 146 to the track planning system 200.

[0044] Route data 146 is the output generated by the route planning system 144, which characterizes how the autonomous vehicle should navigate within a future timeframe in order to follow a specified route.

[0045] For example, route data 146 may reflect a decision by the planning system 144 that, in order to meet certain criteria, the autonomous vehicle should follow the route characterized by route data 146.

[0046] The routes characterized by route data 146 could be, for example, high-level intended routes for an autonomous agent, embedded in an environmental road graph that does not have a concept of time.

[0047] Specific criteria may include, for example, one or more sets of traffic laws (e.g., speed limits, distance, lane crossings, etc.), one or more sets of safety rules (e.g., minimum following distance, speed adjustments for given weather conditions, etc.), or one or more sets of target criteria (e.g., intended destination, minimum deviation from a template route when faced with dynamic road conditions, etc.).

[0048] Route data 146 can be represented in one of several ways.

[0049] For example, route data 146 may be represented by an ordered set of points having a vector direction (e.g., x, y position with a vector direction) relative to the intended route. Each point may represent a waypoint spatial position in a set of equally spaced waypoint spatial positions, each constrained to be set on a route that crosses the center of any given driving lane, for example, a feature present in a road graph, for example, a route represented in a road graph.

[0050] As another example, route data 146 can be represented as a natural language description of the route being traversed.

[0051] As yet another example, route data 146 can be represented as a sequence of one or more vectors, each of which identifies a driving operation (e.g., go straight, turn left, reverse, stop).

[0052] The trajectory planning system 200 processes scene data 142 and route data 146 to generate a final planned trajectory 152. The final planned trajectory 152 characterizes the future trajectory of the autonomous vehicle after the present time.

[0053] Next, the track planning system 200 provides the final planned track 152 to the controller 160 of the autonomous vehicle 102.

[0054] The controller 160 is hardware, software, or a combination of hardware and software mounted on the autonomous vehicle 102 that controls the autonomous vehicle 102. In other words, the controller 160 provides input to various control systems of the autonomous vehicle 103, such as the braking system, steering system, and throttle system, in order to control the movement of the autonomous vehicle 102 in the environment.

[0055] Therefore, the controller 160 maps the final planned track 152 to a series of controls in the vehicle 102's control system and provides control to the control system to control the vehicle 102.

[0056] For example, controller 106 may provide control inputs to the control system at the corresponding time until a new planned track is received, or until a signal is received from another component of the onboard system 110 indicating that controller 160 should stop controlling vehicle 102 using the final track 152.

[0057] To generate the final orbit 152, system 200 uses a predictive neural network.

[0058] The predictive neural network is a neural network configured to process route data 146 and scene data 142 to generate a set of one or more candidate planned trajectories for the vehicle 102 as output. The predictive neural network also generates a set of predicted future trajectories for each set of target agents in the current scene.

[0059] Next, system 200 generates a final trajectory 152 using candidate planned trajectories generated by the predictive neural network, and optionally, predicted future trajectories for the set of target agents.

[0060] For example, system 200 may select one of the candidate planned trajectories and, optionally, post-process the selected candidate planned trajectory to generate the final trajectory 152. Generally, system 200 may generate the final trajectory 152 by applying a set of rules, such as machine learning rules, fixed rules, or a combination of both, to the candidate planned trajectories. In some cases, one or more of the rules may depend on the predicted future trajectory. As an example, one or more of the rules may enforce a minimum distance between the final trajectory 152 and the predicted future trajectory 164 for a set of target agents.

[0061] The generation of the predictive neural network and the final trajectory 152 is described below with reference to Figure 2.

[0062] In some implementations, the onboard system 110 uses the control system 200 to control the vehicle throughout the entire operation of the vehicle 102 in the environment. In some other implementations, the onboard system 110 uses the control system 200 to control the vehicle only in specific driving scenarios, and in other driving scenarios, uses one or more other onboard systems to generate inputs to the controller 160. In a particular embodiment, the onboard system 110 can use the control system 200 for highway or road driving scenarios, while using a different control system for navigating surface roads. In another particular embodiment, the onboard system 110 can use the control system 200 for night driving scenarios, while using a different control system for daytime driving scenarios.

[0063] To generate orbit 152, the orbit planning system 200 may use trained parameter values ​​195, i.e., trained model parameter values ​​of the predictive neural network obtained from the orbit planning model parameter storage unit 190 in the training system 120.

[0064] The training system 120 may train a predictive neural network using any of various imitation learning techniques, such as behavior cloning, adversarial imitation learning, or DAgger (data aggregation) imitation learning from driving logs generated by other autonomous vehicles or manually driven vehicles. As another example, the training system 120 may train a predictive neural network through reinforcement learning, for example, by controlling one or more simulated vehicles in a driving environment simulation. As yet another example, the training system 120 may first train a predictive neural network through imitation learning and then fine-tune the predictive neural network through reinforcement learning.

[0065] The training system 120 is typically hosted within a data center 124, which may be a distributed computing system with hundreds or thousands of computers in one or more locations.

[0066] The training system 120 includes a training data store 170 that stores training data used to train the trajectory planning system, i.e., used to determine the trained parameter values ​​195 of the trajectory planning system 200. The training data store 170 receives raw training examples from agents operating in the real world. For example, the training data store 170 may receive raw training examples 155 from agents, e.g., manually driven vehicles or autonomous vehicles controlled using different planning systems. The raw training examples 155 can be processed by the training system 120 to generate new training examples. The raw training examples 155 may include scene data and route data, such as scene data 142 and route data 146, which can be used as input for new training examples. The raw training examples 155 may also include result data characterizing the state of the environment surrounding the autonomous vehicle 102 at one or more future points in time. Using this resulting data, a ground truth trajectory for the autonomous vehicle at a point in time characterized by the scene data can be generated, and a ground truth trajectory targeting an agent in the scene at that point in time can also be generated. The ground truth trajectory for a given agent, i.e., the target agent, or the autonomous vehicle, identifies the actual trajectory (derived from the resulting data) that the agent will traverse at a future point in time. For example, the ground truth trajectory can identify the spatial position in the autonomous vehicle's center coordinate system that moves the agent at each of several future points in time. Using the resulting data, a ground truth output can be generated for each additional output produced by the predictive neural network, as described later.

[0067] The training data store 170 provides training example 175 to the training engine 180, which is also hosted on the training system 120. The training engine 180 uses training example 175 to update the model parameters used by the trajectory planning system 200 and provides the updated model parameters 185 to the trajectory planning model parameter store 190. For example, the training engine 180 may train a predictive neural network on training example 175 using any suitable imitation learning objective, e.g., one of the objectives described above. If the training example also includes the respective ground truth output for additional outputs, the objective may also measure the error between the additional outputs and the respective ground truth output, e.g., mean squared error, cross-entropy error, etc.

[0068] Once the parameter values ​​of the orbital planning system 200 are fully trained, the training system 120 may transmit the trained parameter values ​​195 to the orbital planning system 200, for example, via a wired or wireless connection.

[0069] Figure 2 is a flowchart of an exemplary process 200 for controlling an autonomous vehicle. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a trajectory planning system (e.g., trajectory planning system 200 in Figure 1), appropriately programmed according to this specification, may perform process 200.

[0070] The system acquires scene data that characterizes the scene in the environment, including the autonomous vehicle and one or more agents, at the present moment (step 202). For example, the system may acquire scene data from sensor measurements of one or more sensors of the autonomous vehicle, and optionally from other sources, such as a road graph of the environment.

[0071] The system receives route data specifying the intended route for the autonomous vehicle after the current time (step 204). As described above, the route data may include data characterizing the intended route for the autonomous vehicle after the current time. For example, route data characterizing the intended route for the autonomous vehicle may be represented in any of the following ways, for example, as an ordered series of locations along the route in natural language, or as a sequence of one or more vectors, each characterizing a driving operation.

[0072] The system uses a predictive neural network to process path data and scene data to generate a predictive output (step 206). Generally, the predictive output includes both (i) a set of one or more candidate planned trajectory outputs specifying each planned trajectory for the autonomous vehicle, starting from the present moment, and (ii) a set of one or more predicted future trajectory outputs specifying each predicted future trajectory for each of the target agent sets, starting from the present moment.

[0073] For example, each candidate planned trajectory output may include data defining the planned future state of the autonomous vehicle for each of several future time steps. As described above, the vehicle's "state" at a time step generally includes the vehicle's position and optionally may include other information such as the vehicle's orientation and speed. The data defining each planned future state may include regressed future states or parameters of a distribution over possible future states, such as a Gaussian distribution. For example, the parameters may include the mean of the distribution and optionally the covariance of the distribution.

[0074] Alternatively, or in addition, each candidate planned trajectory output may include a probability score representing the estimability that each candidate planned trajectory is the optimal trajectory for an autonomous vehicle, given the intended path for the autonomous vehicle.

[0075] Similarly, for each target agent, the predicted future trajectory output may include data defining the predicted future state of each target agent at each of several future time steps. The data defining each predicted future state may include regressed future states or parameters of a distribution over possible future states, such as a Gaussian distribution. For example, the parameters may include the mean of the distribution and, optionally, the covariance of the distribution.

[0076] Alternatively, or in addition, the predicted future trajectory output may include a probability score representing the predicted likelihood that each predicted future trajectory is the actual trajectory trajectory traversed by the target agent.

[0077] For example, the target agents could be all other agents in the scene, or a suitable subset of agents in the scene. For instance, the target agents could include a fixed number of agents closest to the autonomous vehicle, or a fixed number of agents selected by another system as the most important agents for planning the autonomous vehicle's future trajectory.

[0078] Therefore, a predictive neural network processes a single input and generates both candidate planned trajectories for an agent and predicted future trajectories for other agents in parallel.

[0079] Optionally, the predictive output may also generate one or more additional outputs that can be used to control the autonomous vehicle.

[0080] For example, within the prediction output, each candidate planned trajectory output may specify the gear state of the autonomous vehicle at each future time step, given that the autonomous vehicle is following each planned trajectory. Generally, the gear state of the vehicle at a given time step refers to which gear the vehicle's transmission is engaged in at that time step. Examples of gears include first gear, second gear, drive, reverse, neutral, etc.

[0081] As another example, within the prediction output, each candidate planned trajectory output may specify the turn signal state for the autonomous vehicle at each future time step of each planned trajectory, given that the autonomous vehicle is following each planned trajectory. Generally, the vehicle's turn signal state at a given time step identifies whether each of the vehicle's turn signals is active at that time step.

[0082] As another example, the predictive output may also include, for each of one or more traffic signals in the scene, the predicted state of each traffic signal at each of one or more future points in time after the present. The onboard system can then use this predicted state in controlling the vehicle. Examples of traffic signal states include whether the traffic signal is lit, whether it is flashing, and the color of the traffic signal.

[0083] The autonomous vehicle's control system may then use one or more of these additional outputs when a candidate planned trajectory is selected to be used to control the trajectory, for example, as part of determining the appropriate gear for the vehicle at any given time, or as part of determining whether to activate a turn signal at any given time. Alternatively, one or more of the additional outputs may be used to improve the training signals provided to the planner neural network during training, which may then not be generated or may be ignored at estimated time.

[0084] The system selects the final planned trajectory for the autonomous vehicle from the predicted output (step 208). For example, the system may perform this selection as described above.

[0085] The system controls the autonomous vehicle using the final planned trajectory (step 210). That is, the system can provide the final planned trajectory to the controller for the autonomous vehicle, which can then convert the trajectory into a control input for the autonomous vehicle's control system, and apply the control input to move the autonomous vehicle within the environment.

[0086] Figure 3 shows an example 300 of the predictive neural network architecture.

[0087] As shown in the embodiment of Figure 3, the predictive neural network includes an encoder neural network 320 and a decoder neural network 330.

[0088] The encoder neural network 320 processes the scene data to generate an encoded representation of the scene data. The encoded representation of the scene data generally consists of a set of one or more vectors of numerical values ​​representing the scene data.

[0089] In the embodiment shown in Figure 3, the encoder neural network 320 receives inputs as pre-processed feature tensors for different token modalities. These different token modalities may include, for example, agent states for autonomous vehicles and other agents in the environment, static road graph features, dynamic road graph features, and so on.

[0090] The encoder neural network 320 may have any suitable architecture that enables it to map scene data to an encoded representation. For example, the encoder neural network 320 may include any of the following: convolutional layers, fully connected layers, recurrent layers, transformer layers, multi-context gating layers, etc.

[0091] As one embodiment, the encoder neural network 320 may include (i) a separate encoder for each of several different types of scene data, and (ii) a coupled neural network that maps the output of each encoder to an encoded representation. An example of such an architecture is described in detail in U.S. Patent Application No. 17 / 396,554, “AGENT TRAJECTORY PLANNING USING NEURAL NETWORKS,” filed on 6 August 2021, the contents of which are incorporated herein by reference in their entirety.

[0092] The decoder neural network 330 processes the encoded representations of the scene data and path data to generate a prediction output.

[0093] Generally, each set of predicted trajectory outputs is adjusted for the encoded representation of the scene data but not for the path data, while the set of planned trajectory outputs is adjusted for both the encoded representation of the scene data and the path data.

[0094] More specifically, in the embodiment shown in Figure 3, the decoder neural network 330 is configured to maintain the respective queries corresponding to each candidate planned trajectory output and each predicted trajectory output.

[0095] For example, the queries corresponding to each trajectory can be learned queries. These queries are called "learned" because they are learned during the training of the predictive neural network and are independent of the scene data and path data.

[0096] As another example, a query corresponding to a given trajectory may be determined based on scene data, the intended path, or both. For instance, a query corresponding to a given trajectory might represent the current state of the corresponding agent, the type of the corresponding agent (i.e., whether the corresponding agent is an autonomous vehicle or a different target agent), and so on.

[0097] As yet another example, queries corresponding to a given trajectory can be generated by combining learned queries with data generated from scene data, intended paths, or both.

[0098] To process the encoded representations of the scene data and path data and generate a predictive output, the decoder neural network 330 updates each query and then generates the corresponding output from the updated queries.

[0099] For example, the decoder neural network 330 may include a shared predictive head that is applied to each query to generate a corresponding output. The shared predictive head can generally have any suitable neural network architecture. For example, the predictive head may be a multilayer perceptron (MLP) or may include a single linear neural network layer, optionally followed by a nonlinear activation layer. If the predictive neural network generates one or more of the additional outputs described above, the predictive head is shared across all queries, and therefore the predictive head also generally generates additional outputs for queries corresponding to predicted trajectories that the system may discard or otherwise ignore. Having a shared predictive head that is applied to both queries corresponding to planned trajectories and queries corresponding to predicted trajectories simplifies the architecture of the neural network and improves its computational efficiency, because the neural network has fewer parameters than would be required to implement different predictive heads for different types of queries.

[0100] In particular, the decoder neural network 330 updates queries for any other candidate plan trajectories, queries for predicted trajectories, encoded representations of scene data, and queries for any given candidate planner trajectories, adjusted to the intended path. The decoder neural network 330 then generates candidate plan trajectories from the updated queries, for example, by processing the updated queries using a shared prediction head.

[0101] For each predicted future trajectory, the decoder neural network 330 updates the query corresponding to the predicted trajectory, and the query corresponding to the predicted future trajectory, which is adjusted based on the encoded representation of the scene data, but not on the intended path or queries corresponding to other candidate planned trajectories. In this way, the decoder neural network 330 prevents the predicted future trajectory from being generated and adjusted based on privileged information unknown to the corresponding target agent, i.e., the intended future behavior of the autonomous vehicle. The decoder neural network 330 then generates the predicted future trajectory from the updated query, for example, by processing the updated query using a shared prediction head.

[0102] In a specific embodiment, the decoder neural network 330 may include a self-attention layer and a cross-attention layer.

[0103] Each cross-attention layer updates each query by performing cross-attention to the output of the encoder neural network. That is, each attention head of a given cross-attention layer applies an attention mechanism to generate queries, and keys and values, from the encoder output.

[0104] Each self-attention layer updates each query by performing self-attention across the query. That is, each attention head of a given self-attention layer applies an attention mechanism that generates the query, key, and value from the query.

[0105] To ensure that queries for trajectory prediction are not updated based on queries for planned trajectories, the attention mechanism applied by the self-attention layer can be masked to mask autonomous vehicle queries, i.e., queries corresponding to planned trajectories for autonomous vehicles, from agent queries, i.e., queries corresponding to predicted future trajectories for other agents. In particular, autonomous vehicle queries can be allowed to attend to all queries, while agent queries can be restricted to attending only agent queries and not autonomous vehicle queries.

[0106] Generally, each attention head in each self-attention layer generates a set of query vectors, a set of key vectors, and a set of value vectors from the query received as input by the self-attention layer, and then performs self-attention on the query by applying one of various variants of query key-value (QKV) attention, such as a dot product attention mechanism or a scaled dot product attention mechanism, using the query vectors, key vectors, and value vectors to generate an output.

[0107] As a specific example, in the attention head of a self-attention neural network layer, the attention mechanism may be configured to apply query transformations, key transformations, and value transformations to each query to derive the respective query vector, key vector, and value vector for each query used to determine the output. The query transformations, key transformations, and value transformations can be any respective linear transformations or any other suitable learned transformations.

[0108] For example, the attention head may generate updated embeddings for each query by calculating a weighted sum of value vectors, weighted by a similarity function of the query vectors, for queries on the corresponding key vectors. The similarity function could be, for example, the dot product, cosine similarity, or other similarity measures.

[0109] When calculating weights for each agent query to perform masking, the attention head may mask out autonomous vehicle queries by setting the weight corresponding to each autonomous vehicle query to zero in the weighted sum described above. For autonomous vehicle queries, the attention head may refrain from masking any of the weights in the weighted sum.

[0110] As shown in Figure 3, the system may also use a feature preprocessor 310 that processes scene data and path data to generate inputs for the encoder neural network 320 and the decoder neural network 330.

[0111] For example, to tailor queries to the autonomous vehicle along the intended route, the feature preprocessor 310 may generate embeddings representing the intended route by processing the intended route data using, for example, an embedding neural network trained together with a predictive neural network. The embeddings representing the intended route can then be combined with, for example, each query to the autonomous vehicle (but without any queries to other agents) or averaged before being processed by the first layer of the decoder neural network 330.

[0112] Similarly, the feature preprocessor 310 may also process the current state and agent type for each agent, i.e., the autonomous vehicle and each target agent, and generate an embedding for each agent by processing the current state and agent type using, for example, an embedded neural network trained together with a predictive neural network. The resulting embeddings can then be used as queries for the corresponding agents (i.e., before the embedding representing the autonomous vehicle is applied), or they can be combined with, for example, learned embeddings, for example, by being added to or averaged to generate the corresponding agents (i.e., before the embedding representing the autonomous vehicle is applied).

[0113] As another example, the feature preprocessor 310 may preprocess the scene data before it is processed by the encoder neural network 320.

[0114] The predictive neural network may also include one or more output post-processors 340 that post-process the output of the predictive head into corresponding trajectories. For example, the output post-processors 340 may transform the trajectories into a common coordinate system, or, if each trajectory output includes parameters of a probability distribution, may sample or select trajectories from a distribution.

[0115] Figure 4 is a flowchart of an exemplary process 400 for processing input using a shared predictive neural network. For convenience, process 400 is described as being carried out by one or more computer systems located at one or more locations. For example, a trajectory planning system appropriately programmed according to this specification (e.g., trajectory planning system 200 in Figure 1) may carry out process 400.

[0116] The system acquires scene data that characterizes the scenes in the environment and routes data that specifies the intended path for the autonomous vehicle (step 402).

[0117] The system processes the scene data using an encoder neural network of a shared predictive neural network to generate an encoded representation of the scene data (step 404).

[0118] The system processes (i) a representation of the path data and (ii) an encoded representation of the scene data using a decoder neural network of a shared predictive neural network to generate a predictive output (step 406). As described above, the predictive output includes both (i) a set of one or more candidate planned path outputs specifying each planned path for the autonomous vehicle, starting from the present moment, and (ii) a set of one or more predicted future path outputs specifying each predicted future path for each target agent in the set of target agents, starting from the present moment.

[0119] As part of this process, the decoder neural network processes the encoded representation and the representation of the path data to generate their respective representations for each candidate planned trajectory output and each predicted future trajectory output (step 408). For example, as described above, the system maintains each query for each output, and each representation may be an updated query generated by processing each query through a sequence of neural network layers.

[0120] Next, the system uses a shared output head to process the respective representations for each candidate planned trajectory output and each predicted future trajectory output to generate the candidate planned trajectory outputs and predicted future trajectory outputs (step 410).

[0121] This specification uses the term “configured” in relation to systems and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform that operation or action. One or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0122] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more modules of computer program instructions, i.e., computer program instructions encoded on tangible non-temporary storage media to be executed by or to control the operation of a data processing device. The computer storage media may be a machine-readable storage device, a machine-readable storage board, a random-access memory device, or a serial-access memory device, or one or more combinations thereof. Alternatively, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to code information for transmission to a suitable receiving device for execution by a data processing device.

[0123] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for data processing, including, for example, a programmable processor, a computer, or multiple processors or computers. A device may also be, or even include, an application-specific logic circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, a device may also include code that generates the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or a combination of one or more of these.

[0124] Computer programs, also called and written as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, as standalone programs or included as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required, correspond to a file in a file system. A program may be stored in other programs or as part of a file that holds data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple interconnected files, for example, one or more modules, subprograms, or files that store parts of code. Computer programs may be deployed to run on one computer, located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0125] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0126] The processes and logical flows described herein may be implemented by one or more programmable computers that run one or more computer programs that perform functions by acting on input data and producing outputs. The processes and logical flows may also be implemented by application-specific logic circuits (e.g., FPGAs or ASICs) or by a combination of application-specific logic circuits and one or more programmed computers.

[0127] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to receive data from or transmit data to or both. However, a computer is not required to have such devices. Furthermore, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0128] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0129] To provide user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, a keyboard, and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. User interaction may also be provided using other types of devices, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, verbal, or tactile input. Furthermore, the computer may interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to the web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, for example, a smartphone running a messaging application, and receiving response messages from the user.

[0130] Data processing devices for implementing machine learning models may include, for example, application-specific hardware accelerator units for handling the common and computationally intensive parts of machine learning training or production (i.e., inference) workloads.

[0131] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0132] Embodiments of the subject matter described herein may be implemented in a computing system including a backend component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described herein), or in a computing system including one or more such backend, middleware, or frontend components in any combination. The components of the system may be interconnected by digital data communications of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0133] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device, for example, to display data to a user interacting with the device, and to receive user input from that user. Data generated on the user device, such as the results of user interactions, may be received by the server from the device.

[0134] This specification includes many specific implementation details, which should be interpreted not as limitations on the scope of any invention or claimable scope, but as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments or in any preferred partial combination. Furthermore, features may be described above as acting in a particular combination, and may even be initially claimed as such, but in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0135] Similarly, while the operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order, or that all of the operations shown be performed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0136] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the operations described in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily have to be in the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. It is a method, The autonomous vehicle acquires scene data that characterizes the scene in the environment at the present moment, including the autonomous vehicle and a set of target agents. The autonomous vehicle receives route data characterizing the intended route that passes through the environment of the autonomous vehicle after the current time, The autonomous vehicle processes the route data and the scene data using a predictive neural network to generate a predictive output that includes: (i) a set of one or more candidate plan trajectory outputs specifying each planned trajectory for the autonomous vehicle, starting from the current time; and (ii) a set of one or more predicted future trajectory outputs specifying each predicted future trajectory for each target agent in the set of target agents, starting from the current time. Methods that include...

2. The method according to claim 1, further comprising controlling the autonomous vehicle using the predictive output.

3. The method according to claim 1 or 2, wherein the scene at the present time includes a plurality of other agents, and the set of target agents is a subset of the plurality of other agents.

4. The method according to claim 3, wherein the subset is an appropriate subset.

5. The method according to any one of claims 1 to 4, wherein each candidate planned trajectory output includes data defining each planned future state of the autonomous vehicle at each of a plurality of future time steps.

6. The method according to any one of claims 1 to 5, wherein each candidate planned trajectory output further includes a probability score representing the likelihood that each of the candidate planned trajectories is the optimal trajectory for the autonomous vehicle, given the intended path.

7. The method according to any one of claims 1 to 6, wherein, for each of the target agents, each predicted future trajectory output includes data defining each predicted future state of the target agent in each of the multiple future time steps.

8. The method according to any one of claims 1 to 7, further comprising a probability score representing the predicted probability that each of the predicted future trajectories is an actual trajectory traversed by the target agent.

9. The aforementioned predictive neural network, An encoder neural network that processes the aforementioned scene data to generate an encoded representation of the aforementioned scene data, A decoder neural network that processes the encoded representations of the scene data and the path data to generate the prediction output, The method according to any one of claims 1 to 8, comprising:

10. Each set of the predicted trajectory outputs is adjusted for the encoded representation of the scene data, but not for the path data. The method according to claim 9, wherein the set of planned trajectory outputs is adjusted for the encoded representation of the scene data and the path data.

11. The decoder neural network is configured to maintain the respective queries corresponding to each candidate planned trajectory output and each predicted trajectory output, and to process the encoded representations of the scene data and the path data to generate the predicted output. For each of the above candidate planned trajectories, Updating the query corresponding to the candidate planner trajectory, adjusted based on the query corresponding to any other candidate plan trajectory, the query corresponding to the predicted trajectory, the encoded representation of the scene data, and the intended path, From the updated query, generate the candidate planned trajectory, The method according to claim 10, including the method described in claim 10.

12. The process of the encoded representations of the scene data and the path data to generate the prediction output is as follows: For each of the predicted future trajectories, The query corresponding to the predicted future trajectory, adjusted based on the query for the corresponding encoded representation of the scene data, is updated, but not based on the query for the intended path or other candidate planned trajectories. From the updated query, generate the predicted future trajectory, The method according to claim 11, including the method described in claim 11.

13. The method according to any one of claims 9 to 12, wherein the decoder neural network, which processes the encoded representations of the scene data and the path data to generate the prediction output, includes a shared prediction head that generates each candidate planned trajectory output and each predicted future trajectory output.

14. The method according to any one of claims 1 to 13, wherein each candidate planned trajectory output specifies, in each future time step of each planned trajectory, the autonomous vehicle is following each planned trajectory, and the autonomous vehicle is following each planned trajectory.

15. The method according to any one of claims 1 to 14, wherein each candidate planned trajectory output specifies, in each future time step, the gear state for the autonomous vehicle, given that the autonomous vehicle is following each of the respective planned trajectories.

16. The method according to any one of claims 1 to 15, wherein the prediction output further includes, for each of the one or more traffic signals in the scene, the predicted state of each of the traffic signals at each of the one or more future points in time after the present time.

17. A system comprising one or more computers and one or more storage devices that store instructions causing the one or more computers to perform each of the operations described in any of the prior claims when executed by the one or more computers.

18. One or more computer-readable media for storing instructions that cause one or more computers to perform each of the operations described in any one of claims 1 to 16 when executed by one or more computers.

Images