Ranking agents near an autonomous vehicle by mutual importance

Abstract

Methods, systems and apparatus for identifying high priority agents around a vehicle, including a computer program encoded on a computer storage medium. The high priority agents may be identified based on a set of mutual importance scores, wherein each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent in the set of agents to the planning decisions of the other. The mutual importance scores may be calculated based on importance scores evaluated from the perspective of the vehicle and the agents.

Description

Technical Field

[0001] This specification relates to autonomous vehicles. Background Art

[0002] Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use a variety of onboard sensors and computer systems to detect nearby objects and use these detections to make control and navigation decisions. Summary of the invention

[0003] This specification describes a system implemented as a computer program on a vehicle that generates prediction data for an agent around the vehicle based on an estimated level of interaction between the agent and the vehicle. Specifically, the estimated level of interaction between the vehicle and the agent can reflect not only the possible impact that the agent will have on the planning decision of the vehicle, but also the possible impact that the vehicle will have on the planning decision of the agent.

[0004] In a first aspect, an embodiment of the subject matter disclosed herein includes a method performed by a system having one or more data processing devices. The method may include identifying a set of agents around a vehicle. The system determines a first set of importance scores and one or more second sets of importance scores. Each importance score in the first set indicates an estimated relevance of a different agent in the set of agents to a planning decision of the vehicle. Each second set is determined by specifying a different agent in the set of agents as a reference agent for the second set, and each importance score in each second set indicates an estimated relevance of a different non-reference agent or vehicle to a planning decision of the reference agent. A set of mutual importance scores is then determined based on the first set of importance scores and the one or more second sets of importance scores, wherein each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent in the set of agents to another's planning decision. At least one score in the set of mutual importance scores can be used in a process of planning the movement of the vehicle.

[0005] These and other implementations may also include one or more of the following features.

[0006] Determining the first set of importance scores may include processing first environmental data representing a portion of an environment surrounding the vehicle, and the first environmental data may be oriented around the vehicle.

[0007] The first environmental data may identify the location of the set of agents in the environment surrounding the vehicle.

[0008] Determining the first set of importance scores may also include processing first motion data characterizing movement of the vehicle. The first motion data may describe at least one of a past position of the vehicle in the environment, a current position of the vehicle in the environment, a predicted future position of the vehicle in the environment, a past speed of the vehicle, a current speed of the vehicle, a predicted future speed of the vehicle, a past acceleration of the vehicle, a current acceleration of the vehicle, a predicted future acceleration of the vehicle, a past heading of the vehicle, a current heading of the vehicle, or a predicted future heading of the vehicle.

[0009] Determining the first set of importance scores may also include processing second motion data characterizing movement of one or more agents in the set of agents around the environment.

[0010] A first importance scoring model may be used to determine a first set of importance scores, and a second importance scoring model may be used to determine one or more second sets of importance scores. The first importance scoring model may be more computationally demanding than the second importance scoring model. The first importance scoring model and the second importance scoring model may include respective neural networks (e.g., feedforward neural networks, recursive neural networks, or convolutional neural networks).

[0011] Determining a first set of importance scores may include designating a vehicle as a reference agent and designating a set of agents surrounding the vehicle as corresponding non-reference agents.

[0012] For each second set, determining one or more second sets of importance scores may include: processing second environmental data characterizing a portion of an environment surrounding a corresponding reference agent of the second set. The second environmental data may be oriented around the corresponding reference agent of the second set.

[0013] Determining a first set of importance scores may include processing first motion data representing at least one of past movement or predicted future movement of a vehicle. One or more second sets of importance scores may be determined without processing motion data representing past movement or predicted future movement of a set of vehicles or agents.

[0014] The set of mutual importance scores may indicate a higher mutual importance to a first agent, wherein the first agent has (i) a higher importance to the vehicle's planning decision according to the first set of importance scores, and (ii) the vehicle has a higher importance to the first agent's planning decision according to a second set of importance scores corresponding to the first agent. The set of mutual importance scores may indicate a lower mutual importance to a second agent, wherein the second agent has (i) a lower importance to the vehicle's planning decision according to the first set of importance scores, and (ii) the vehicle has a lower importance to the second agent's planning decision according to the second set of importance scores corresponding to the second agent.

[0015] The set of mutual importance scores may be probability values ​​indicating an absolute level of mutual importance, or may be ranking values ​​indicating how the agents rank relative to each other with respect to mutual importance to the vehicle.

[0016] The vehicle can be fully autonomous or semi-autonomous, and the collection of agents can include other vehicles, pedestrians, or cyclists around the vehicle.

[0017] Determining a first set of importance scores may include processing a feature representation of each of the set of agents, wherein the feature representation of the agent includes one or more of: a speed of the agent, an acceleration of the agent, a type of the agent, a distance of the agent from a vehicle, and data indicating whether a predicted trajectory of the agent will overlap with a trajectory of the vehicle.

[0018] An appropriate subset of the set of agents having the highest mutual importance scores may be identified as high priority agents. Data characterizing the agents may be generated using the first prediction model only for those agents in the set of agents identified as high priority agents. Using at least one score in the set of mutual importance scores in a process for planning movement of the vehicle may include providing data characterizing the high priority agents generated using the first prediction model to a planning system of the vehicle to generate a planning decision for planning a future trajectory of the vehicle.

[0019] In additional aspects, embodiments of the subject matter disclosed herein include a system comprising a data processing device and a memory in communication with the data processing device, the memory storing instructions, wherein the instructions, when executed, cause the data processing device to perform actions corresponding to any of the methods disclosed in this specification. In additional aspects, embodiments of the subject matter disclosed herein include one or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by the data processing device, cause the data processing device to perform actions corresponding to any of the methods disclosed in this specification.

[0020] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0021] In order for the planning system of the vehicle to generate planning decisions that cause the vehicle to travel along a safe and comfortable trajectory, timely and accurate prediction data (e.g., behavior prediction data) of the agents around the vehicle (e.g., pedestrians, other motor vehicles, cyclists) should be provided to the planning system. However, generating prediction data for the agents around the vehicle may require a large amount of computing resources (e.g., memory, computing power, or both). In some cases, a large number of other agents in the environment may be around the vehicle (e.g., when the vehicle is traveling along a busy highway). In these cases, the limited computing resources available on the vehicle may not be sufficient to generate timely prediction data for all agents around the vehicle.

[0022] The in-vehicle system described in this specification can focus the limited computing resources (e.g., memory, computing power, or both) available on the vehicle to generate accurate prediction data only for those agents that are considered to be the most important (such as those agents that are considered to be the most interactive with the vehicle or those agents that have the greatest impact on the planning decisions generated by the vehicle's planning system). Alternatively, predictions of different resolutions, types, or degrees can be made for different agents based on the level of importance of each agent. In this way, the in-vehicle system described in this specification can generate timely prediction data that enables the planning system to generate planning decisions that allow the vehicle to travel along a safe and comfortable trajectory regardless of the limited computing resources available on the vehicle. In addition, this specification describes a technique for improving the identification of high-priority agents by evaluating the mutual importance of agents in the vehicle's surrounding environment. By evaluating the mutual importance of agents, the vehicle can more accurately determine which agents exhibit an interactive relationship with the vehicle. For example, the mutual importance of agents can be based not only on the estimated relevance of the agent to the vehicle's planning decision (from the vehicle's perspective), but also on the estimated relevance of the vehicle to the agent's planning decision (from the agent's perspective).

[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram of an example in-vehicle system and an example training system for training an importance scoring model.

[0025] Figure 2 is a block diagram of an example importance scoring system and an example prediction system.

[0026] Figure 3A Depicted are example scores of a reference agent by the primary and secondary scoring models.

[0027] Figure 3B Depicted is the process flow for generating mutual importance scores for the agents surrounding a vehicle.

[0028] Figure 4 Depicts an example data pipeline in an agent-centric data representation that can be processed by an importance scoring model.

[0029] Figure 5A-Figure 5B is a flow chart of an example process for generating mutual importance scores and using the mutual importance scores to select high priority agents for which to generate prediction data.

[0030] Figure 6 is a flow chart of an example process for generating training examples that can be used to train an importance scoring model.

[0031] The same reference numbers and designations in different drawings indicate the same elements. DETAILED DESCRIPTION

[0032] This specification describes how an onboard system of a vehicle generates prediction data (e.g., behavior prediction data) for agents around the vehicle. The agents around the vehicle may be, for example, pedestrians, cyclists, or other vehicles. In order to generate prediction data, the onboard system determines the corresponding mutual importance scores of all or at least a subset of the detected agents around the vehicle. The mutual importance scores of the agents characterize the estimated level of interaction between the vehicle and the agents from the perspective of the vehicle and the agents. For example, the mutual importance of the agents can be evaluated by first estimating the relevance (e.g., influence) of the agent on the planning decisions of the vehicle (e.g., decisions related to planning the future trajectory of the vehicle). The system can then estimate the relevance of the vehicle to the agent's own planning decisions from the standpoint of the agent. Using the estimates from the perspective of the vehicle and the agent, a mutual importance score reflecting the estimated mutual relevance or interaction between the vehicle and each agent can be determined. The onboard system identifies which agents around the vehicle are "high priority" agents based at least on the mutual importance scores, and generates accurate prediction data for these high priority agents using a prediction model. For the remaining agents around the vehicle that are not high priority agents, the onboard system can use a less computationally intensive (but potentially less accurate) prediction model to generate prediction data. Alternatively, the onboard system can avoid generating prediction data for some or all of the agents that are not determined to be high priority agents. These and other features are described in more detail below.

[0033] Figure 1is a block diagram of an example in-vehicle system 100. The in-vehicle system 100 is comprised of hardware and software components, some or all of which are physically located on the vehicle 102. In some cases, the in-vehicle system 100 may generate planning decisions to plan a future trajectory for the vehicle 102, present information to a driver of the vehicle 102 to assist the driver in safely operating the vehicle, or both. For example, in response to determining that the trajectory of another vehicle may intersect the trajectory of the vehicle 102, the in-vehicle system 100 may generate a planning decision to adjust the future trajectory of the vehicle 102 to avoid a collision (e.g., by braking). As another example, in response to determining that the trajectory of another vehicle may intersect the trajectory of the vehicle 102, the in-vehicle system 100 may present a warning message that prompts the driver of the vehicle 102 to adjust the trajectory of the vehicle 102 to avoid a collision.

[0034] although Figure 1 The vehicle 102 in the embodiment is depicted as a driving car, and the examples in this specification are described with reference to a driving car, but in general the vehicle 102 may be any kind of vehicle. For example, in addition to a driving car, the vehicle 102 may be a boat or an airplane. In addition, the vehicle system 100 may include Figure 1 Components other than those depicted in (e.g., a collision detection system or a navigation system).

[0035] The vehicle-mounted system 100 includes a sensor system 104 that enables the vehicle-mounted system 100 to "see" the environment around the vehicle 102. More specifically, the sensor system 104 includes one or more sensors, some of which are configured to receive reflections of electromagnetic radiation from the environment around the vehicle 102. For example, the sensor system 104 may include one or more laser sensors (e.g., LIDAR laser sensors) configured to detect reflections of laser light. As another example, the sensor system 104 may include one or more radar sensors configured to detect reflections of radio waves. As another example, the sensor system 104 may include one or more camera sensors configured to detect reflections of visible light.

[0036] The sensor system 104 continuously (i.e., at each of a plurality of time points) captures raw sensor data that may indicate the direction, intensity, and distance traveled by reflected radiation. For example, a sensor in the sensor system 104 may transmit one or more pulses of electromagnetic radiation in a particular direction, and may measure the intensity of any reflections and the time at which the reflections were received. The distance may be calculated by determining the time that elapsed between transmitting a pulse and receiving its reflection. Each sensor may continuously scan a particular space in angle, azimuth, or both. Scanning in azimuth may, for example, allow a sensor to detect multiple objects along the same line of sight.

[0037] At any given point in time, one or more agents in the environment may be around the vehicle 102. The agents around the vehicle 102 may be, for example, pedestrians, cyclists, or other vehicles. The onboard system 100 uses a prediction system 106 to continuously (e.g., at each of a plurality of time points) generate prediction data 108 characterizing some or all of the agents around the vehicle 102. For example, for each of the plurality of agents around the vehicle 102, the prediction data 108 may be behavior prediction data defining the corresponding probability of the agent making each of a predetermined number of possible driving decisions (e.g., yielding, changing lanes, passing, braking, or accelerating). As another example, for each of the plurality of agents around the vehicle 102, the prediction data 108 may be classification prediction data defining the corresponding probability of the agent being each of a predetermined number of possible agent types (e.g., animals, pedestrians, cyclists, cars, or trucks).

[0038] The in-vehicle system 100 may provide the prediction data 108 generated by the prediction system 106 to the planning system 110 , the user interface system 112 , or both.

[0039] When the planning system 110 receives the prediction data 108, the planning system 110 may use the prediction data 108 to generate a planning decision 144 to plan a future trajectory for the vehicle 102. The planning decisions 144 generated by the planning system 110 may include, for example: yielding (e.g., to pedestrians), stopping (e.g., at a "Stop" sign), passing other vehicles, adjusting the vehicle lane position to accommodate bicyclists, slowing down in schools or construction zones, merging (e.g., onto a highway), and stopping. In a specific example, the onboard system 100 may provide the planning system 110 with behavior prediction data indicating that the future trajectory of another vehicle may intersect the future trajectory of the vehicle 102, potentially resulting in a collision. In this example, the planning system 110 may generate a planning decision 144 to apply the brakes of the vehicle 102 to avoid the collision.

[0040] The planning decision 144 generated by the planning system 110 may be provided to a control system 142 of the vehicle 102. The control system 142 of the vehicle may control some or all operations of the vehicle by implementing the planning decision 144 generated by the planning system. For example, in response to receiving the planning decision 144 to apply the brakes of the vehicle, the control system 142 of the vehicle 102 may send an electronic signal to a brake control unit of the vehicle. In response to receiving the electronic signal, the brake control unit may mechanically apply the brakes of the vehicle.

[0041] When the user interface system 112 receives the prediction data 108, the user interface system 112 may use the prediction data 108 to present information to the driver of the vehicle 102 to help the driver safely operate the vehicle 102. The user interface system 112 may present the information to the driver of the vehicle 102 in any suitable manner, such as an audio message sent through a speaker system of the vehicle 102, or through a warning displayed on a visual display system in the vehicle (e.g., an LCD display on the dashboard of the vehicle 102). In a specific example, the in-vehicle system 100 may provide the user interface system 112 with prediction data 108 indicating that the future trajectory of another vehicle may intersect with the future trajectory of the vehicle 102, potentially resulting in a collision. In this example, the user interface system 112 may present a warning message to the driver of the vehicle 102 with instructions to adjust the trajectory of the vehicle 102 to avoid the collision.

[0042] In order for the planning system 110 to generate planning decisions that cause the vehicle 102 to travel along a safe and comfortable trajectory, the onboard system 100 must provide timely and accurate prediction data 108 to the planning system 110. However, the prediction system 106 may require a large amount of computing resources (e.g., memory, computing power, or both) to generate prediction data 108 for the agents around the vehicle. In some cases, a large number of other agents in the environment may be around the vehicle 102 (e.g., when the vehicle 102 is traveling along a busy highway). In these cases, the limited computing resources available on the vehicle 102 may not be sufficient for the prediction system 106 to generate timely prediction data 108 for all agents around the vehicle 102.

[0043] In order to enable the prediction system 106 to generate timely prediction data 108, the onboard system 100 may identify one or more of the agents around the vehicle 102 as "high priority" agents. In some cases, the onboard system 100 only identifies a portion of the agents around the vehicle 102 as high priority agents. The prediction system 106 is then configured to generate accurate prediction data 108 for the high priority agents using a prediction model (e.g., a prediction neural network). The prediction system 106 may generate prediction data 108 for any remaining agents that are not identified as high priority agents (which will be referred to as "low priority" agents in this specification) using a less computationally intensive (but potentially less accurate) behavior prediction model. Alternatively, the onboard system 100 may avoid generating any prediction data 108 for some or all of the low priority agents. In this specification, a "prediction model" should be understood as implementing a prediction algorithm.

[0044] The onboard system 100 uses the importance scoring system 114 and the ranking engine within the prediction system 106 to determine which agents around the vehicle 102 are designated as high priority agents. Figure 2 5 , the importance scoring system generates a set of importance scores 116 indicating the estimated relevance of agents in the environment on the vehicle, or other estimated relevance of non-reference agents in the environment on a designated reference agent. The importance scoring system may generate the importance scores 116 using one or more scoring models, which may be implemented as, for example, a neural network model, a random forest model, a support vector machine (SVM) model, or any other type of trainable machine learning model.

[0045] In some examples, a primary importance scoring model and a secondary importance scoring model are provided within the system 114, and each model is configured to process input data representations of environmental data 148 and, optionally, motion (e.g., trajectory) data 150. Environmental data 148 may include current observations of the environment based on sensor data 118, predefined data such as road maps, and indications of the current positions of the vehicle and secondary agents in the environment. Motion data 150 may describe past motion parameters, future planned motion parameters, or predicted motion parameters, or both, of the vehicle, the secondary agent, or both. Importance scores 116 are processed to determine mutual importance scores between the vehicle and each secondary agent. Based on the mutual importance scores, the in-vehicle system 100 may determine one or more secondary agents with the highest importance scores around the vehicle as high priority agents.

[0046] In a particular example, at a particular intersection there may be: (i) an oncoming vehicle that may turn into the path of vehicle 102, and (ii) a cyclist that is far behind vehicle 102 and is unlikely to influence the planning decisions of planning system 110. In this example, importance scoring system 114 may generate a higher importance score for the oncoming vehicle than for the cyclist, potentially resulting in the oncoming vehicle being designated as a high priority agent and the cyclist being designated as a low priority agent.

[0047] By generating prediction data 108 based on mutual importance scores, the onboard system 100 can focus its limited computing resources on generating accurate prediction data 108 for only those agents that are likely to have the greatest impact on the planning decisions generated by the planning system 110. In this way, the onboard system 100 can generate timely prediction data 108 that enables the planning system 110 to generate planning decisions that cause the vehicle to travel along a safe and comfortable trajectory despite the limited computing resources available on the vehicle 102.

[0048] In some cases, the onboard system 100 may provide the mutual importance scores directly to the planning system 110. The planning system 110 may use the mutual importance scores by, for example, allocating a greater amount of computing resources to process data representing high-priority agents than to process data representing low-priority agents when generating planning decisions 144. More generally, any processing module of the onboard system (including, but not limited to, the prediction system 106 and the planning system 110) may use the mutual importance scores to allocate a greater amount of computing resources to process data representing high-priority agents than to process data representing low-priority agents.

[0049] To allow importance scoring system 114 to accurately prioritize agents, training system 120 may determine training parameter values ​​122 for an importance scoring model included in importance scoring system 114. Training system 120 is typically hosted within a data center 124, which may be a distributed computing system having hundreds or thousands of computers in one or more locations.

[0050] The training system 120 includes a training importance scoring model 126 that is configured to process the agent feature representations to generate corresponding importance scores 140 for the agents. The training system 120 includes multiple computing devices with software or hardware modules that implement the operations of the training importance scoring model 126. For example, if the training importance scoring model 126 is an importance scoring neural network, the training system 120 includes multiple computing devices with software or hardware modules that implement the corresponding operations of each layer of the training importance scoring neural network according to the architecture of the training importance scoring neural network. The training importance scoring model 126 is typically the same model (or nearly the same model) as the on-board importance scoring model. For example, if the training importance scoring model is a neural network, then it typically has (at least in part) the same architecture as the on-board importance scoring neural network included in the importance scoring system 114.

[0051] The training importance scoring model 126 may calculate operations of the training importance scoring model 126 using current values ​​128 of parameters stored in a set of model parameters 130. Although shown as logically separate, the model parameters 130 and the software or hardware modules that perform the operations may actually be located on the same computing device or the same memory device.

[0052] The training system 120 trains the training importance scoring model 126 using a training dataset 132 including a plurality of training examples 134. The training examples 134 may be obtained from real or simulated driving logs. Each of the training examples 134 includes a corresponding feature representation of each agent surrounding the vehicle at a corresponding point in time, and a label defining a ranking of the agents in order of their influence on the planning decision generated by the vehicle's planning system. Figure 6 An example process for generating training examples 134 is described.

[0053] The training importance scoring model 126 may process the corresponding agent feature representations from the training examples 134 to generate corresponding importance scores 140 for each training example 134. The training engine 136 analyzes the importance scores 140 and compares the rankings of the agents defined by the importance scores 140 with the rankings of the agents defined by the labels in the training examples 134. For example, when the training importance scoring model 126 is a neural network, the training engine 136 may calculate the gradient of a loss function that characterizes the difference between the rankings of the agents defined by the generated importance scores 140 and the rankings of the agents defined by the labels in the training examples 134. In a specific example, the loss function may be a binary (e.g., cross entropy) loss function that characterizes whether the rankings of two agents defined by the generated importance scores 140 are the same as the rankings of the two agents defined by the labels in the training examples 134.

[0054] The training engine 136 then generates updated model parameter values ​​138 by using an appropriate machine learning training technique (e.g., stochastic gradient descent). The training engine 136 can then use the updated model parameter values ​​138 to update the set of model parameters 130.

[0055] After training is complete, the training system 120 may provide the set of training parameter values ​​122 to the in-vehicle system 100 for generating importance scores that enable generation of timely and accurate behavior prediction data 108. The training system 120 may provide the set of training parameter values ​​122 via a wired or wireless connection to the in-vehicle system 100.

[0056] Figure 2 is a block diagram of an example importance scoring system 114 and an example prediction system 106. Importance scoring system 114 and prediction system 106 are examples of systems implemented as computer programs on one or more computers at one or more locations that implement the systems, components, and techniques described below.

[0057] The importance scoring system 114 is configured to process initial environment data 148 characterizing the environment 202 around the vehicle 102 and initial motion data 150 characterizing the movement of the vehicle 102 (and optionally, auxiliary agents in the environment) to generate a set of importance scores 116. A respective set of importance scores 116 may be generated for each agent in the environment around the vehicle 102 (including the vehicle 102 as a primary agent and nearby auxiliary agents). The set of importance scores 116 for each agent (i.e., a reference agent) includes scores that each indicate an estimated influence on a planning decision of the planning system as a result of the presence of another agent (i.e., a non-reference agent) in the environment on the reference agent of the set 116.

[0058] exist Figure 2 , the environment 202 is a roadway, and two vehicles, represented as agent A and agent B, are around vehicle 102. Agent A is maneuvering to cut in front of vehicle 102, while agent B is traveling parallel to vehicle 102. In this example, importance scoring system 114 may assign a higher importance score to agent A than to agent B, specifically because agent A may have a greater influence on the planning decisions generated by planning system 110 of vehicle 102. For example, planning system 110 of vehicle 102 may generate a planning decision that causes vehicle 102 to brake to avoid a collision with agent A.

[0059] To generate the set of importance scores 116, the importance scoring system 114 processes the initial environment data 148 and the initial motion data 150 using the data preparation engine 206. In some embodiments, the data preparation engine 206 is configured to identify each of the auxiliary agents surrounding the vehicle 102 and generate a corresponding agent-centric data input 222 or 224 for each agent. Figure 3A-3B and Figure 4 Additional details regarding techniques for generating agent-centric data representations 222 and 224 are described.

[0060] In some embodiments, the agent-centric data representation 222, 224 may include a feature representation of each non-reference agent around a reference agent in the environment. The feature representation of the agent may be digital data represented in any appropriate digital format (e.g., as a vector or matrix) that characterizes the agent. The feature representation of the agent may include one or more of the following: the speed of the agent (e.g., measured in miles per hour - mph), the acceleration of the agent (e.g., measured in feet per second squared), the type of the agent (e.g., pedestrian, vehicle, cyclist, etc.), the distance of the agent to the reference agent (e.g., measured in feet), data indicating whether (and when) the predicted future trajectory of the agent will overlap with the future trajectory of the reference agent, the position of the agent relative to the reference agent, and data characterizing the speed, acceleration, and heading of the agent within a previous time window (e.g., 1, 2, 5, or 10 seconds).

[0061] The primary importance scoring model 210a processes the primary agent-centric data representation 222 to generate a first set of importance scores describing the estimated impact of each secondary agent on the planning decision of the vehicle 102. The secondary importance scoring model 210b processes each secondary agent-centric data representation 224 to generate a second set of importance scores describing the estimated impact of other (non-reference) agents on the planning decision of the corresponding secondary agent of the set. The importance scoring models 210a, 210b can be implemented as any suitable machine learning model, such as a neural network (with any suitable architecture), a random forest, or a support vector machine. Reference Figure 3A-3B Additional details regarding the importance scoring models 210a, 210b are described.

[0062] The importance score set 116 generated by the importance scoring system 114 can be used by the prediction system 106 to generate prediction data 108 that characterizes one or more agents around the vehicle. Specifically, the importance score set 116 can be processed by the ranking engine 212 to determine the estimated level of interaction between the vehicle 102 and each auxiliary agent around the vehicle 102. The estimated level of interaction can be calculated by evaluating the estimated relevance of each agent to the planning decision of the vehicle 102, and vice versa, and the estimated level can be expressed as a mutual importance score. The prediction data 108 derived in part using the mutual importance score can be behavior prediction data that characterizes the predicted future behavior of one or more of the agents around the vehicle 102. In this example, for one or more of the agents around the vehicle 102, the prediction data 108 can define the corresponding probability of each of the agents making a predetermined number of possible driving decisions (e.g., giving way, changing lanes, overtaking, braking, or accelerating). As another example, prediction data 108 may be classification prediction data defining respective probabilities that the agent is each of a predetermined number of possible agent types (eg, an animal, a pedestrian, a cyclist, a car, or a truck).

[0063] The prediction system 106 also uses the ranking engine 212 to identify which (if any) auxiliary agents around the vehicle 102 are high priority agents. For example, the ranking engine 212 can classify each auxiliary agent with a mutual importance score that meets a predetermined threshold as a high priority agent. In a specific example, the ranking engine can identify each agent with a mutual importance score exceeding a predetermined threshold of 0.8 as a high priority agent. As another example, the ranking engine 212 can identify a predetermined number of auxiliary agents with the highest importance scores as high priority agents. In a specific example, the ranking engine 212 can identify three agents with the highest mutual importance scores as high priority agents. As another example, the ranking engine can identify a dynamically determined number of agents with the highest mutual importance scores as high priority agents based on the computing resources currently available on the vehicle for behavior prediction.

[0064] After identifying the high priority agents 214, the prediction system 106 uses the high priority prediction model 216 to generate corresponding prediction data 108 for each of the high priority agents 214. The high priority prediction model 216 can generate the prediction data 108 for the agents by, for example, processing a representation of the trajectory of the agents up to the current point in time using a predictive neural network.

[0065] The prediction system 106 may use the low-priority prediction model 218 to generate corresponding prediction data 108 for each of the remaining agents (i.e., the low-priority agents 220) that the ranking engine 212 did not identify as the high-priority agents 214. In some cases, the prediction system 106 processes each of the low-priority agents 220 with the same low-priority prediction model. In some cases, the prediction system 106 includes multiple low-priority prediction models with different numbers of model parameters, and processes low-priority agents with higher importance scores with low-priority behavior prediction models with more model parameters. Typically, the high-priority prediction model 216 has a larger number of model parameters than the low-priority prediction model 218, and therefore can generate more accurate prediction data 108 than the low-priority prediction model 218. However, the low-priority prediction model 218 can consume less computing resources (e.g., memory, computing power, or both) than the high-priority prediction model 216. In a particular example, the high priority prediction model 216 may be implemented as a deep neural network having thousands of model parameters, while the low priority prediction model 218 may be implemented as a linear model having tens or hundreds of model parameters.

[0066] In some cases, the prediction system 106 may avoid generating prediction data 108 for some or all of the low-priority agents 220. For example, the prediction system 106 may defer generating behavior prediction data for the low-priority agents 220 until a later point in time when more computing resources are available. In these examples, the prediction system 106 may require that prediction data be generated for each of the agents surrounding the vehicle (regardless of their priority) at at least a minimum frequency (e.g., once per second).

[0067] The prediction data 108 generated by the high priority prediction model 216 for the high priority agent 214 and any prediction data 108 generated by the low priority prediction model 218 for the low priority agents are provided to the planning system 110 for use in generating planning decisions.

[0068] refer to Figure 3A, shows a pair of example scoring processes for generating importance scores for agents surrounding a vehicle (e.g., a self-driving car or other autonomous vehicle). In the first example, the primary agent is the self-driving car (black shaded area), and around the self-driving car, five other vehicles are identified as auxiliary agents (white shaded areas). The first data input set 302a describes information about the environment surrounding the self-driving car, and may also include information describing the past movements of the self-driving car (e.g., moves), past movements of the auxiliary agents, predicted movements or planned movements of the self-driving car, and / or predicted movements of the auxiliary agents. In some embodiments, the first data input set 302 includes a cascade of multiple data channels, such as Figure 4 In addition, the first data input set 302a is centered on the master agent and is thus oriented around the master agent (i.e., the self-driving car). Specifically, the agent around which the data input set is oriented is referred to herein as the "reference" agent, and Figure 3A The self-driving car as the reference agent is represented by a star symbol overlaid on the representation of the vehicle.

[0069] In some embodiments, orienting a data input set around a reference agent involves focusing the data input set so that the reference agent is located at or near the center of a two-dimensional patch representing an environmental area. For example, the data input set 302a may include one or more layers corresponding to different data channels (e.g., road map data, position data, speed data, heading data, acceleration data). Each layer may be aligned with other layers and include a two-dimensional grid, wherein each position or cell in the grid represents a corresponding position in a real-world environment. The reference agent may be substantially at the center of the 2D grid, while the non-reference agent may occupy a grid position surrounding the reference agent. The data input set may also be oriented to match the heading of the reference agent, such as so that the reference agent always points to the top of the grid. In some embodiments, in the case where the data input set describes the features of the auxiliary agent instead of the grid representation, or the data input set also describes the features of the auxiliary agent in addition to the grid representation, feature values ​​may be calculated with respect to the reference agent. For example, the distance or heading of a non-reference agent may be calculated relative to the current position and heading of the reference agent.

[0070] The first data input set 302a is processed by the primary importance scoring model 210a. In some embodiments, the primary importance scoring model 210a is reserved for processing inputs where the primary agent (e.g., a self-driving car) is a reference agent. The primary importance scoring model 210a may have more computational requirements than the secondary importance scoring model 210b, as it can process larger, more informative input data sets and needs to perform more operations to generate outputs. For example, the primary importance scoring model 210a can be configured to process not only environmental data representing the current state of the environment surrounding the reference agent, but also historical motion data and / or predicted motion data for the reference agent and / or non-reference agents. In contrast, the secondary importance scoring model 210b can be configured to process more primitive inputs (e.g., a vector of feature values ​​for each non-reference agent rather than a multi-layer two-dimensional input), inputs that describe the current state of the environment but without past motion data or future motion data for the reference agent and / or non-reference agent, or inputs that describe the current state of the environment and past motion data for the reference agent and / or non-reference agent but without future motion data. Providing separate scoring models 210a and 210b can be advantageous by allowing for more refined generation of importance scores when the primary agent is the reference agent, without significantly increasing the computational cost associated with determining importance scores when the secondary agent is the reference agent. In addition, it may not be easy for an autonomous vehicle to have the same range of data inputs available for scoring from the perspective of the secondary reference agent as for scoring from the perspective of the primary reference agent. Nonetheless, if the data inputs available for the primary and secondary reference agents are relatively equivalent, the same importance scoring model can be employed to process the inputs in both cases.

[0071] The main importance scoring model 210a processes the first data input set 302a to generate a first set 304a of importance scores. The first set 304a of importance scores indicates the estimated relevance or influence of each auxiliary agent on the planning decision of the driverless car, and is generally determined independently of the mutual influence that the driverless car has on the planning decision of the auxiliary agent. The score can be in the form of a value indicating an absolute estimate (e.g., a likelihood or probability value), or in the form of a ranking value indicating the relative importance of the agent relative to other agents around the driverless car. For example, auxiliary agent 3 is immediately behind the driverless car in the same lane, and therefore, auxiliary agent 3 affects the planning decision about the future movement of the driverless car to a greater extent than auxiliary agent 1, where auxiliary agent 1 is traveling in the opposite direction in a lane on the opposite side of the road. Therefore, it is estimated that agent 3 has greater importance than agent 1.

[0072] The second data input set 302b describes information about the environment surrounding the auxiliary agent 4. The second data input set 302b may or may not include information about the past movement or predicted movement of the auxiliary agent 4 (i.e., the reference agent), the past movement or predicted movement of non-reference agents (e.g., self-driving cars and agents 3, 5-6), or both. The second data input set 302b is centered on the auxiliary agent and is therefore oriented around the auxiliary agent (i.e., agent 4) as the reference agent. The auxiliary importance scoring model 210b processes the second data input set 302b to generate a second set 304b of importance scores. The second set 304b of importance scores indicates the estimated relevance or influence of each non-reference agent around agent 4 on the planning decision of the reference agent (i.e., agent 4), and is generally determined independently of the mutual influence that the reference agent has on the planning decision of the non-reference agent. For example, agent 4 is shown in this scenario as being flanked by agents 3 and 5, and as a result, agents 3 and 5 are estimated to have a greater influence on the planning decisions of agent 4 than the self-driving car or agent 6.

[0073] continue Figure 3A In the example, Figure 3B , a process for generating mutual importance scores based on a set of (non-mutual) importance scores centered on an agent is depicted in . The importance scoring system 114 generates a first set 304a of importance scores by processing an input 302a centered on a self-driving car with a primary importance scoring model 210a. The system 114 generates additional sets of importance scores 304b-304n by processing data inputs 304b-304n centered on secondary agents with secondary importance scoring models 210b. The system 114 then provides each set of importance scores 304a-304n to a ranking engine 212. The ranking engine 212 is programmed to process the corresponding set of importance scores 304a-304n to generate a set of mutual importance scores 306. The mutual importance scores indicate an estimated level of mutual correlation between a primary agent (i.e., a self-driving car) and all or some of the secondary agents detected around the primary agent. Mutual correlation is a two-way measure of the interaction between a pair of agents. Unlike the agent-centric set of importance scores 304a-n that reflect the estimated influence of non-reference agents on the planning decision of the reference agent (independent of the mutual consideration of the influence of the reference agent on the planning decision of the non-reference agent), mutual correlation takes into account the influence on the planning decision in both directions.

[0074] The ranking engine 212 may use any suitable function to generate mutual importance scores 306. Typically, when the importance score of a particular auxiliary agent centered on the driverless car is high and the importance score of the driverless car from the perspective of the particular auxiliary agent is also high, the mutual correlation between the driverless car and the particular auxiliary agent will be the highest. When the importance score from the perspective of the driverless car, the importance score from the perspective of the particular auxiliary agent, or both are lower, the mutual correlation decreases. In some embodiments, the ranking engine 212 generates the corresponding mutual importance scores between the driverless car and each auxiliary agent by taking the sum of the importance scores between the driverless car and the auxiliary agent from the perspective of the driverless car (i) and (ii) the importance scores between the driverless car and the auxiliary agent from the perspective of the auxiliary agent. The summed scores may be based on absolute importance scores or ranking values. In some embodiments, when taking the sum of the importance scores between the driverless car and the particular auxiliary agent to determine the mutual importance score, the score from the perspective of the driverless car may be weighted higher than the score from the perspective of the auxiliary agent. In some embodiments, the ranking engine 212 generates mutual importance scores 306 by starting with the SDC-centric importance scores 304a and updating the rankings of auxiliary agents in the first set of scores 304a by increasing the ranking of any auxiliary agent whose agent-centric importance score / ranking relative to the autonomous vehicle is sufficiently higher (e.g., at least a threshold ranking value).

[0075] Figure 4 is a block diagram of an example data preparation system 206. Data preparation system 206 is an example of a system implemented as a computer program on one or more computers at one or more locations that implements the systems, components, and techniques described below.

[0076] The data preparation system 206 processes the initial environment data 148 and the initial motion data 150 to generate a data representation 222 centered around a primary agent. Although a data representation oriented around a primary agent (e.g., a self-driving car) is shown here, the data preparation system 206 can similarly prepare an agent-centric data representation for secondary agents surrounding the primary agent. The data preparation system 206 receives an indication of which agent represented in the initial environment data 148 and the initial motion data 150 should be a reference agent, and then prepares a data representation oriented around the identified reference agent. In some cases, fewer data channels can be processed when generating an agent-centric data representation for a secondary agent than when the system 206 generates a data representation 222 centered around the primary agent.

[0077] The data representation 222 consists of multiple "channels". Each channel is a two-dimensional (2D) array of data values ​​representing the "top-down" angle of the environment around the vehicle (i.e., the reference agent, in this example, the master agent (self-driving car)). Each position in each channel corresponds to a corresponding spatial position in the environment and can be indexed by corresponding coordinates. Each of the channels has the same dimensions (i.e., the same number of rows and columns), the same top-down angle of the environment, and is aligned (i.e., registered) with each other. That is, positions indexed by the same coordinates in different channels correspond to the same spatial position in the environment. In other words, corresponding positions in different channels correspond to the same spatial position in the environment.

[0078] To represent the trajectory of the vehicle, the data representation 222 includes a vehicle time channel and a corresponding vehicle motion parameter channel corresponding to each of a predetermined number of motion parameters. Each of the motion parameters characterizes a corresponding characteristic of the vehicle's motion, such as the vehicle's speed, acceleration, or heading. Figure 4 As depicted, data representation 222 may include vehicle time channel 402 , vehicle speed channel 404 , vehicle acceleration channel 406 , and vehicle heading channel 408 .

[0079] The vehicle time channel and the vehicle motion parameter channel represent the previous trajectory of the vehicle up to the current time point, and optionally, represent candidate future trajectories of the vehicle after the current time point. The candidate future trajectory of the vehicle refers to a possible trajectory of the vehicle after the current time point. More specifically, the candidate future trajectory of the vehicle specifies, for each of a plurality of future time points after the current time point, a spatial position in the environment that may be occupied by the agent at the future time point. For each of the plurality of future time points, the candidate future trajectory also specifies a possible value for each of a predetermined number of motion parameters that characterize the motion of the vehicle at the future time point.

[0080] For the sake of brevity, the following description relates to a vehicle time channel and a vehicle motion parameter channel, which represent the previous trajectory of the vehicle up to the current time point. It should be understood that in addition to the previous trajectory of the vehicle up to the current time point, the vehicle time channel and the vehicle motion parameter channel can also similarly represent the candidate future trajectory of the vehicle after the current time point.

[0081] The vehicle time channel represents corresponding points in time when the vehicle occupied different spatial positions in the environment in the vehicle's previous trajectory. Each motion parameter channel represents the value of a corresponding motion parameter that characterizes the motion of the vehicle when the vehicle occupied different spatial positions in the environment in the vehicle's previous trajectory. Specifically, for each spatial position occupied by the vehicle in the environment, the data value in the vehicle time channel corresponding to the spatial position defines the time point when the vehicle occupied the spatial position. In addition, the data value in each corresponding vehicle motion parameter channel corresponding to the spatial position defines the value of the corresponding motion parameter that characterizes the motion of the vehicle when the vehicle occupied the spatial position in the environment.

[0082] Typically, the environment may include multiple spatial locations that are not included in the vehicle's previous trajectory or candidate future trajectory. For these spatial locations, the data preparation system 206 may set the data values ​​in the vehicle time channel and the vehicle motion parameter channel corresponding to these spatial locations to default values ​​(e.g., value 0).

[0083] When the same spatial position in the environment is occupied by the vehicle at multiple time points, the system 206 can set the corresponding data values ​​in the vehicle time channel and the motion parameter channel corresponding to the spatial position to any appropriate value. For example, the system 206 can set the corresponding data values ​​in the vehicle time channel and the motion parameter channel corresponding to the spatial position according to the last time point when the vehicle occupied the spatial position. In other words, the system 206 can set the data value corresponding to the spatial position in the vehicle time channel to define the last time point when the vehicle occupied the spatial position. In addition, the system 206 can set the corresponding data value in the vehicle motion parameter channel to define the value of the corresponding motion parameter that characterizes the motion of the vehicle when the vehicle last occupied the spatial position.

[0084] To represent the trajectory of the agent in the environment around the vehicle, the data representation optionally includes an auxiliary time channel and a corresponding auxiliary motion parameter channel corresponding to each of a predetermined number of motion parameters. Figure 4 As depicted, data representation 222 may include auxiliary time channel 410 , auxiliary velocity channel 412 , auxiliary acceleration channel 414 , and auxiliary heading channel 416 .

[0085] The auxiliary time channel and the auxiliary motion parameter channel jointly represent the previous trajectory of the non-reference agent (in this example, all auxiliary agents) up to the current time point, and optionally represent the predicted future trajectory of the non-reference agent after the current time point. The system 206 can obtain the predicted future trajectory of the agent from the previous time point as the behavior prediction output of the behavior prediction neural network. The predicted future trajectory of the agent specifies the spatial position in the environment that can be occupied by the agent at the future time point for each of a plurality of future time points after the current time point. For each of the plurality of future time points, the predicted future trajectory also specifies the possible values ​​of each of a predetermined number of motion parameters that characterize the motion of the agent at the future time point.

[0086] For brevity, the following description refers to auxiliary time channels and auxiliary motion parameter channels, which represent the previous trajectory of the agent up to the current time point. It should be understood that in addition to the previous trajectory of the agent up to the current time point, the auxiliary time channels and auxiliary motion parameter channels can also similarly represent the predicted future trajectory of the agent after the current time point.

[0087] The auxiliary time channels jointly represent corresponding time points at which the agent occupied different spatial positions in the environment in the agent's previous trajectory. Each auxiliary motion parameter channel represents the value of a corresponding motion parameter that characterizes the movement of the agent when the agent occupied different spatial positions in the environment in the agent's previous trajectory. Specifically, for each spatial position in the environment occupied by one of the agents, the data value corresponding to the spatial position in the auxiliary time channel defines the time point at which the agent occupied the spatial position. In addition, the data value corresponding to the spatial position in each corresponding auxiliary motion parameter channel defines the value of the corresponding motion parameter that characterizes the movement of the agent when the agent occupied the spatial position in the environment.

[0088] Typically, the environment may include multiple spatial locations that are not included in any agent's previous trajectory or predicted future trajectory. For these spatial locations, the data preparation system 206 may set the data values ​​corresponding to these spatial locations in the auxiliary time channel and the auxiliary motion parameter channel to default values ​​(e.g., value 0).

[0089] When the same spatial location in the environment is occupied by agents at multiple points in time, the system 206 may set the corresponding data values ​​in the auxiliary time channel and the auxiliary motion parameter channel corresponding to the spatial location to any appropriate values. For example, the system 206 may set the corresponding data values ​​in the auxiliary time channel and the auxiliary motion parameter channel corresponding to the spatial location based on the last point in time that one of the agents occupied the spatial location.

[0090] By jointly representing the trajectory of the agent using a single auxiliary time channel and a single auxiliary motion parameter corresponding to each motion parameter, the system 206 can generate a data representation having a predetermined dimension that is independent of the (variable) number of agents. In this way, the data representation 222 generated by the system 110 can be easily processed by an importance scoring model that is configured to process the data representation 222 of the predetermined dimension.

[0091] In some embodiments, in addition to the time channel and the motion parameter channel, the system 206 may include additional channels in the data representation 222. For example, the data representation 222 may include one or more of the following: a road map channel, a vehicle positioning channel, and an auxiliary positioning channel. The road map channel represents the known geometry of the environment surrounding the vehicle 102. For example, the road map channel may represent the locations of different features of the environment, including: roads, different lanes in the road, crosswalks, traffic lights, construction areas, school areas, etc. The vehicle positioning channel represents the spatial position of the vehicle in the environment at the current point in time (e.g., defined by a bounding box). The auxiliary positioning channels (e.g., auxiliary positioning channels 418) jointly represent the position of each of the agents in the environment at the current point in time (e.g., defined by a bounding box). Figure 4 In the example depicted in , the road map channel and the vehicle positioning channel are superimposed onto a single channel 420 .

[0092] As depicted at 422, the system 110 determines a data representation 222 by aligning and channel-wise concatenating the generated channels. That is, the data representation 222 is a concatenation of the generated channels. By implicitly representing the respective trajectories of the vehicle and the agent in this format, the system 206 generates a representation that is compact and can be efficiently processed by the importance scoring model 210a or 210b. In some examples, the importance scoring model 210a and / or 210b can be a convolutional neural network model. The importance scoring model can process the data representation using a convolutional neural network layer defined by multi-dimensional (e.g., two-dimensional or three-dimensional) convolutional filters, thereby enabling the importance scoring model to learn complex spatial relationships between different spatial regions of the environment. The data representation 222 centered on the master agent corresponds to Figure 2 The same numbers as shown in the data represent, and Figure 3A-3B A first data input 302a is shown centered around the SDC.

[0093] Figure 5A-Figure 5B is a flow chart of an example process 500, 550 for generating mutual importance scores and using the mutual importance scores to select a high priority agent for which to generate prediction data. For convenience, the processes 500, 550 will be described as being performed by a system of one or more computers located in one or more locations. For example, an in-vehicle system (e.g., Figure 1 The vehicle-mounted system 100) can execute processes 500 and 550.

[0094] The system obtains initial environment data and initial motion data (502). Typically, the initial environment and motion data have not been formatted for processing by the importance scoring model. The initial environment data and motion data may or may not be oriented around the main agent (e.g., an unmanned vehicle), but are typically not oriented around the auxiliary agents around the main agent. The initial environment data may include sensor data describing the observation of the environment at the current point in time (e.g., based on LIDAR signals, camera images, etc.). The initial environment data may also include a road map, the spatial position of the main agent in the environment at the current point in time, and the spatial position of the agent in the environment at the current point in time. The initial motion data may include past motion data and / or future motion data of the main agent and / or auxiliary agents in the environment.

[0095] The system identifies secondary agents around the vehicle (primary agent) (504). For example, the system may use an object detection neural network to process aspects of the environmental data to generate data defining the predicted locations and types of multiple objects around the vehicle (e.g., vehicles, pedestrians, cyclists, etc.). The system may then identify the secondary agents around the vehicle as those objects that have been identified as, for example, vehicles, people, or cyclists. In some cases, the system may not identify any agents around the vehicle, for example, if the vehicle is traveling on an otherwise empty roadway. In these cases, the system may not generate any predictive data at that point in time. The description of processes 500, 550 below assumes that the system identifies one or more agents around the vehicle.

[0096] The system selects a reference agent (506). The reference agent is an agent relative to which the presence of other (non-reference) agents is evaluated to determine the possible impact of other agents on the planning decisions of the reference agent. In some cases, the first system may first select a vehicle (main agent) as the reference agent and then select a secondary agent as the reference agent in subsequent iterations. The data preparation engine processes the initial environment and motion data to generate an agent-centric representation of the environment and motion data oriented around the selected reference agent (508). The importance scoring model processes the agent-centric data representation to generate an agent-centric importance score set (510). The agent-centric importance score set indicates the estimated relevance or impact of the non-reference agent on the future planning decisions of the reference agent (e.g., the planning of the trajectory of the future movement of the reference agent) for each non-reference agent around the reference agent. The agent-centric importance score set can be calculated without considering the predicted impact of the reference agent on the planning decisions of the non-reference agent. In some embodiments, a separate importance scoring model is used to process the input data representation depending on whether the reference agent is the main agent or the secondary agent. These operations (506-510) are repeated until a set of agent-centric importance scores have been generated for the primary agent and all identified secondary agents. For example, the system can determine whether additional agents remain to be evaluated (512), and if so, return to 510 to select the remaining agents as the next reference agents. Once a set of agent-centric importance scores has been generated for the primary agent and all secondary agents, the ranking engine can generate a set of mutual importance scores based on the agent-centric importance scores (514). The agent-centric importance scores indicate an estimated level of interaction (e.g., mutual influence or mutual correlation) between the primary agent and each secondary agent. For example, if the independent agent-centric importance scores between the primary agent and a given secondary agent are both high, the ranking engine can assign a high mutual importance score to the secondary agent.

[0097] The mutual importance scores may then be applied by one or more systems to the various endpoints (516). Figure 5BProcess 550 represented by the flowchart in illustrates one such application. For example, another aspect of a ranking engine (e.g., ranking engine 212) or a prediction system (e.g., prediction system 106) can identify a set of high priority auxiliary agents around a vehicle based on mutual importance scores (552). For example, the system can identify each auxiliary agent with a mutual importance score that meets a predetermined threshold as a high priority agent. As another example, the system can identify a predetermined number of auxiliary agents with the highest importance scores as high priority agents. In some cases, the system can identify none or all of the auxiliary agents around the vehicle as high priority agents.

[0098] The system generates prediction data for the high priority agent using the prediction model (554). In some embodiments, the prediction model can generate classification prediction data predicting the type of auxiliary agent (e.g., animal, pedestrian pushing a shopping cart, pedestrian guiding traffic, pedestrian riding a scooter, car, truck, etc.). In some embodiments, the behavior prediction model can generate behavior prediction data characterizing the predicted future behavior of the auxiliary agent (e.g., braking, acceleration, lane change, etc.). In these embodiments, the behavior prediction model can generate behavior prediction data for the auxiliary agent by, for example, processing a representation of the trajectory of the auxiliary agent up to the current point in time using a behavior prediction neural network. The following are a few examples.

[0099] In some embodiments, for each of a plurality of spatial locations in the environment, the behavior prediction data of the agent defines a corresponding probability that the agent will occupy the spatial location at a specified time point after the current time point. In these embodiments, the output layer of the behavior prediction neural network can be a sigmoid layer having corresponding neurons corresponding to each of the spatial locations in the environment. The probability that the agent will occupy a specific spatial location at a specified time point after the current time point can be defined as the activation of a neuron in the sigmoid output layer corresponding to the specific spatial location.

[0100] In some embodiments, the behavior prediction data of the agent defines the probability that the candidate future trajectory of the agent is the actual future trajectory of the agent. In these embodiments, the output layer of the behavior prediction neural network may include a single neuron, wherein the activation of the neuron is defined as the output of an activation function (e.g., a sigmoid activation function) with an output range of [0, 1]. The probability that the candidate future trajectory of the agent is the actual future trajectory of the agent can be defined as the activation of a single neuron in the output layer.

[0101] In some embodiments, the behavior prediction data of the agent defines the corresponding probability that the agent will make each of a predetermined number of possible driving decisions. For example, possible driving decisions may include one or more of the following: yielding, changing lanes, overtaking, braking, and accelerating. In these embodiments, the output layer of the behavior prediction neural network may be a sigmoid layer with corresponding neurons corresponding to each of the possible driving decisions. The probability that the agent will make a specific driving decision after the current time point can be defined as the activation of the neurons in the sigmoid output layer corresponding to the specific driving decision.

[0102] Optionally, the system generates predictive data for any low-priority agents (i.e., secondary agents that are not designated as high-priority agents) using one or more predictive models that are less computationally intensive than the predictive models used to generate predictive data for high-priority agents (556). For example, the predictive model(s) used to generate predictive data for low-priority agents may have fewer model parameters than the predictive model(s) used to generate predictive data for high-priority agents. In a specific example, the predictive model(s) may be a neural network, and the neural network(s) used for the low-priority agents may have fewer neural network weights than the neural network(s) used for the high-priority agents. In some cases, the system may determine not to generate any predictive data for some or all of the low-priority agents (e.g., if their respective importance scores do not meet a predetermined threshold).

[0103] The system provides the generated prediction data (of the high priority agent, and optionally, any low priority agents) to the vehicle's planning system (558). The planning system uses the prediction data to generate planning decisions that plan the future trajectory of the vehicle.

[0104] Figure 6 6 is a flow chart of an example process 600 for generating training examples that can be used to train an importance scoring model. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 1 The training system 120) can perform process 600.

[0105] The system obtains sensor data representing the environment around the vehicle (master agent) at a given point in time (602). The system may obtain the sensor data from a sensor data log stored in a physical data storage device or a logical data storage area. The sensor data may be real sensor data representing the real world environment around the vehicle, or simulated sensor data representing a simulated environment around a simulated vehicle. Figure 1As described, the sensor data may be generated by one or more sensors of the vehicle, including one or more of the following: a laser sensor, a radar sensor, and a camera sensor. The system may also obtain other forms of environmental and motion data as described herein.

[0106] The system generates corresponding prediction data for the auxiliary agents around the vehicle (604). For example, the system can generate corresponding prediction data for each auxiliary agent around the vehicle that is identified by the vehicle's object detection system. The behavior prediction data generated by the system for each agent characterizes the predicted future behavior of the agent. For example, for each of the agents around the vehicle, the behavior prediction data can define the corresponding probability of the agent making each of a predetermined number of possible driving decisions (e.g., yielding, changing lanes, overtaking, braking, or accelerating). To generate the behavior prediction data for the agent, the system can use a behavior prediction model (e.g., reference Figure 5B The behavior prediction neural network described in

[21] is used to process the representation of the agent’s trajectory up to a given point in time.

[0107] The system provides the behavior prediction data to a planning system configured to process the behavior prediction data to generate planning decisions for planning a future trajectory of the vehicle (606). The planning decisions generated by the planning system may include, for example, yielding (e.g., to other vehicles), stopping (e.g., at a stop sign), passing other vehicles, adjusting the vehicle lane position to accommodate bicyclists, slowing down at schools or construction zones, merging (e.g., onto a highway), and stopping.

[0108] The system processes the planning decisions generated by the planning system to determine a ranking of the auxiliary agents around the vehicle at a given point in time in order of their impact on the vehicle's planning decisions (608). The system can determine the ranking of the auxiliary agents based on the extent to which the behavior of each agent causes the planning system to generate planning decisions that will change the future trajectory of the vehicle. For example, the system can determine a score for each agent based on the extent to which each agent causes the planning system to generate planning decisions that will change the future trajectory of the vehicle, and then use the score to determine the ranking of the auxiliary agents. In a specific example, the system can determine that agent A has a score of 12, agent B has a score of 0, and agent C has a score of 6.5, in which case the system can determine the ranking of the agents as: [A, C, B].

[0109] For example, each of the planning decisions may define: (i) an action to be performed by the vehicle (e.g., accelerate, decelerate, stop, or turn), and (ii) the agents around the vehicle that caused the planning system to generate the planning decision (if applicable). In this example, the system may generate a score of 0 for agents that do not cause the planning system to generate any planning decisions. For agents that do cause the planning system to generate planning decisions, the system may determine the score of the agent as a predetermined function of the action specified by the planning decision. For example, if the planning decision specifies an action of braking at 10 feet per second squared, the system may determine the score of the agent as 10. If a single agent causes the planning system to generate multiple planning decisions (e.g., braking and turning), the system may determine the score of the agent as the sum (or other combination) of the corresponding scores generated for each of the multiple planning decisions.

[0110] The ranking of agents may be broadly understood as any data defining certain agents around a vehicle that have a greater influence on a planning decision than certain other agents around the vehicle. For example, the ranking of agents may define an explicit ordering of agents from the highest ranking to the lowest ranking, wherein each agent is assigned a different ranking. As another example, the ranking of agents may define a division of agents into a plurality of groups, wherein each group comprises one or more agents. For each pair of groups comprising a first group and a second group, the ranking may define whether an agent in the first group has a greater influence on a planning decision than an agent in the second group. Agents within a given group may be understood to have an equivalent influence on a planning decision.

[0111] The system generates training examples that include: (i) a corresponding feature representation of each agent surrounding the vehicle, and (ii) label data (610) defining a ranking of the agents surrounding the vehicle. Figure 2 - As described in Figure 5, the system can generate a feature representation of the agent from a portion of the sensor data that characterizes the agent, from behavior prediction data generated for the agent by a behavior prediction system at a previous point in time, or from any other information source.

[0112] In some embodiments, a secondary importance scoring model can be trained similarly. However, the training examples used in this case can be formatted so that the secondary agent is the reference agent in the training sample, and the primary agent (e.g., a self-driving car) and other secondary agents are non-reference agents.

[0113] This specification uses the term "configured" in conjunction with system and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when run by a data processing device, cause the device to perform the operation or action.

[0114] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium, for operation by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, wherein the artificially generated propagation signal is generated to encode information for transmission to a suitable receiver device for operation by a data processing device.

[0115] The term "data processing device" refers to data processing hardware and includes all kinds of devices, equipment and machines for processing data (including, for example, a programmable processor, a computer, or multiple processors or computers). The device may also be or also include a special logic circuit, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may also optionally include code that creates an operating environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0116] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines or code portions). A computer program may be deployed to be run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0117] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed on and run on the same (multiple) computers.

[0118] The processes and logic flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC) or by a combination of a special purpose logic circuit and one or more programmed computers.

[0119] A computer suitable for running a computer program can be based on a general or special microprocessor or both, or a central processing unit of any other kind. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into a special logic circuit. Typically, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data or be operably coupled to receive data from the one or more large-capacity storage devices or to transfer data thereto. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (Personal Digital Assistant, PDA), a mobile audio or video player, a game console, a global positioning system (Global Positioning System, GPS) receiver, or a portable storage device (e.g., a universal serial bus (Universal Serial Bus, USB) flash drive), just to name a few examples.

[0120] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0121] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including sound input, voice input, or tactile input. In addition, the computer may interact with the user by transmitting documents to and receiving documents from a device used by the user; for example, by transmitting a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by transmitting text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in exchange.

[0122] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts (i.e., inference, workloads) of machine learning training or production.

[0123] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0124] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0125] A computing system may include a client and a server. The client and the server are usually remote from each other and generally interact through a communication network. The relationship between the client and the server occurs by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated at a user device, such as a result of a user interaction, may be received from the device at the server.

[0126] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but rather descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the claimed combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0127] Similarly, although operations are depicted in the drawings and set forth in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged as multiple software products.

[0128] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions set forth in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: Identify multiple agents around the vehicle; determining a first set of importance scores, wherein each importance score in the first set indicates an estimated relevance of a different agent among the plurality of agents to a planning decision of the vehicle; determining one or more second sets of importance scores, wherein each second set is determined by assigning a different agent from the plurality of agents as a reference agent for the second set, and each importance score in each second set indicates an estimated relevance of a different non-reference agent or the vehicle to a planning decision of the reference agent; determining a set of mutual importance scores based on the first set of importance scores and one or more second sets of importance scores, wherein each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent of the plurality of agents to a planning decision of the other; identifying a subset of the plurality of agents having the highest mutual importance scores as high priority agents; generating data characterizing the high priority agent using a first predictive model; as well as The data characterizing the high-priority agent generated using the first predictive model is provided to a planning system of the vehicle to generate a planning decision for planning a future trajectory of the vehicle.

2. The method according to claim 1, wherein: Determining the first set of importance scores includes processing first environmental data characterizing a portion of an environment surrounding the vehicle, wherein the first environmental data is oriented around the vehicle.

3. The method according to claim 2, wherein: The first environmental data identifies the locations of the plurality of agents in an environment surrounding the vehicle.

4. The method according to claim 2, wherein: Determining the first set of importance scores further includes processing first motion data characterizing movement of the vehicle.

5. The method according to claim 4, wherein: The first motion data describes at least one of the past position of the vehicle in the environment, the current position of the vehicle in the environment, the predicted future position of the vehicle in the environment, the past speed of the vehicle, the current speed of the vehicle, the predicted future speed of the vehicle, the past acceleration of the vehicle, the current acceleration of the vehicle, the predicted future acceleration of the vehicle, the past heading of the vehicle, the current heading of the vehicle, or the predicted future heading of the vehicle.

6. The method according to claim 4, wherein: Determining the first set of importance scores also includes processing second motion data characterizing movement of one or more agents of the plurality of agents around the environment.

7. The method of claim 1, further comprising determining the first set of importance scores using a first importance scoring model, and determining one or more second sets of importance scores using a second importance scoring model.

8. The method according to claim 7, wherein: The first importance scoring model is more computationally demanding than the second importance scoring model.

9. The method according to claim 7, wherein: The first importance scoring model includes a convolutional neural network.

10. The method according to claim 1, wherein: Determining the first set of importance scores includes designating the vehicle as a reference agent and designating the plurality of agents surrounding the vehicle as corresponding non-reference agents.

11. The method according to claim 1, wherein: Determining one or more second sets of the importance scores comprises, for each second set: Second environmental data characterizing a portion of an environment surrounding a corresponding reference agent of the second set is processed, wherein the second environmental data is oriented around the corresponding reference agent of the second set.

12. The method of claim 1, wherein: Determining the first set of importance scores includes processing first motion data characterizing at least one of past movement or predicted future movement of the vehicle; and The one or more second sets of importance scores are determined without processing motion data characterizing past movement or predicted future movement of the vehicle or the plurality of agents.

13. The method according to claim 1, wherein: The set of mutual importance scores indicates a higher mutual importance to a first agent, wherein the first agent has (i) a higher importance to the planning decision of the vehicle according to the first set of importance scores, and (ii) the vehicle has a higher importance to the planning decision of the first agent with respect to the first agent according to a second set of importance scores corresponding to the first agent.

14. The method according to claim 13, wherein: The set of mutual importance scores indicates a lower mutual importance to a second agent, wherein the second agent has (i) a lower importance to the vehicle's planning decisions based on the first set of importance scores, and (ii) the vehicle has a lower importance to the second agent's planning decisions based on a second set of importance scores corresponding to the second agent.

15. The method according to claim 1, wherein: The set of mutual importance scores are either probability values ​​indicating an absolute level of mutual importance, or ranking values ​​indicating the rankings of agents relative to each other regarding mutual importance to the vehicles.

16. The method according to claim 1, wherein: The vehicle is a fully autonomous or semi-autonomous vehicle, and the plurality of agents include other vehicles, pedestrians, or cyclists around the vehicle.

17. The method according to claim 1, wherein: Determining the first set of importance scores includes processing a feature representation of each of the plurality of agents, wherein the feature representation of the agent includes one or more of: a velocity of the agent, an acceleration of the agent, a type of the agent, a distance of the agent from the vehicle, and data indicating whether a predicted trajectory of the agent will overlap with a trajectory of the vehicle.

18. A system comprising: Data processing equipment; and A memory in communication with the data processing device and storing instructions, wherein when the instructions are executed, the data processing device performs operations, the operations comprising: Identify multiple agents around the vehicle; determining a first set of importance scores, wherein each importance score in the first set indicates an estimated relevance of a different agent among the plurality of agents to a planning decision of the vehicle; determining one or more second sets of importance scores, wherein each second set is determined by assigning a different agent from the plurality of agents as a reference agent for the second set, and each importance score in each second set indicates an estimated relevance of a different non-reference agent or the vehicle to a planning decision of the reference agent; determining a set of mutual importance scores based on the first set of importance scores and one or more second sets of importance scores, wherein each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent of the plurality of agents to a planning decision of the other; identifying a subset of the plurality of agents having the highest mutual importance scores as high priority agents; generating data characterizing the high priority agent using a first predictive model; and The data characterizing the high priority agent generated using the first predictive model is provided to a planning system of the vehicle to generate a planning decision for planning a future trajectory of the vehicle.

19. One or more non-transitory computer-readable storage media storing instructions that, when executed by a data processing device, cause the data processing device to perform operations comprising: Identify multiple agents around the vehicle; determining a first set of importance scores, wherein each importance score in the first set indicates an estimated relevance of a different agent among the plurality of agents to a planning decision of the vehicle; determining one or more second sets of importance scores, wherein each second set is determined by assigning a different agent from the plurality of agents as a reference agent for the second set, and each importance score in each second set indicates an estimated relevance of a different non-reference agent or the vehicle to a planning decision of the reference agent; determining a set of mutual importance scores based on the first set of importance scores and one or more second sets of importance scores, wherein each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent of the plurality of agents to a planning decision of the other; identifying a subset of the plurality of agents having the highest mutual importance scores as high priority agents; generating data characterizing the high priority agent using a first predictive model; as well as The data characterizing the high priority agent generated using the first predictive model is provided to a planning system of the vehicle to generate a planning decision for planning a future trajectory of the vehicle.

Images