Navigation trajectories using reinforcement learning for autonomous vehicles in navigation networks

Abstract

An autonomous vehicle includes a decision maker module and a classifier module coupled to a parser module. The decision maker module generates a trajectory decision for a current time, generates a current two-dimensional slice of a flat space around the autonomous vehicle, generates a future two-dimensional slice of the flat space by projecting the current two-dimensional slice of the flat space forward in time, and generates a three-dimensional state space by stacking the current two-dimensional slice and the future two-dimensional slice. The classifier module generates a reward for the trajectory decision based on the recent behavior of the autonomous vehicle. The parser module selects a final trajectory decision of the autonomous vehicle from the trajectory decisions based on the three-dimensional state space and the reward. The current two-dimensional slice includes a current autonomous vehicle position and a current neighboring vehicle position. The future two-dimensional slice includes a future autonomous vehicle position and a future neighboring vehicle position.

Description

Technical Field

[0001] The present disclosure relates to a system and method for navigating trajectories using reinforcement learning of an ego vehicle in a navigation network. Background Art

[0002] Many existing on-board trajectory planning systems generate a single trajectory for the vehicle's current state and surroundings. However, the state space used to determine the trajectory is usually large, where the input state includes all available visual information and granular output actions such as steering angle and throttle acceleration. In order to train a neural network to determine a single trajectory, many parameters are adjusted and datasets of different scenarios are often used.

[0003] What is desired is a technique for navigating trajectories using reinforcement learning of autonomous vehicles in a navigation network. Summary of the invention

[0004] An autonomous vehicle is disclosed herein. The autonomous vehicle includes a plurality of decision maker modules and a classifier module. The plurality of decision maker modules are coupled to a parser module. The plurality of decision maker modules are configured to: generate a plurality of trajectory decisions at a current time, generate a current two-dimensional slice of a flat space around the autonomous vehicle, generate a plurality of future two-dimensional slices of the flat space around the autonomous vehicle by projecting the current two-dimensional slice of the flat space forward in time, and generate a three-dimensional state space by stacking the current two-dimensional slice and a plurality of future two-dimensional slices. The classifier module is coupled to the parser module. The classifier module is configured to generate a plurality of rewards for the plurality of trajectory decisions based on the most recent behavior of the autonomous vehicle. The parser module is configured to select a final trajectory decision of the autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards. The current two-dimensional slice includes a current self-position of the autonomous vehicle at the current time and a plurality of current neighboring positions of a plurality of neighboring vehicles. The plurality of future two-dimensional slices include a plurality of future self-positions of the autonomous vehicle at a plurality of future time points and a plurality of future neighboring positions of a plurality of neighboring vehicles.

[0005] In one or more embodiments of an autonomous vehicle, the parser module uses reinforcement learning to select the final trajectory decision.

[0006] In one or more embodiments, the autonomous vehicle further includes a control module coupled to the resolver module and configured to navigate the autonomous vehicle in response to the final trajectory decision.

[0007] In one or more embodiments of the autonomous vehicle, the parser module is part of a server computer external to the autonomous vehicle.

[0008] In one or more embodiments, the autonomous vehicle further includes a transmitter configured to transmit the plurality of trajectory decisions to the server computer.

[0009] In one or more embodiments, the autonomous vehicle further includes a receiver configured to receive the final trajectory decision from the server computer.

[0010] In one or more embodiments, the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the autonomous vehicle and the plurality of neighboring vehicles.

[0011] In one or more embodiments, the autonomous vehicle further comprises a memory device configured to store lane information. The plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the lane information.

[0012] In one or more embodiments, the autonomous vehicle further comprises a memory device configured to store traffic light information. The plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the traffic light information.

[0013] The present invention provides a method for navigation trajectory recognition using reinforcement learning. The method includes: using a circuit to generate multiple trajectory decisions of an autonomous vehicle at the current time; generating a current two-dimensional slice of a flat space around the autonomous vehicle; generating multiple future two-dimensional slices of the flat space around the autonomous vehicle by projecting the current two-dimensional slice of the flat space forward in time; generating a three-dimensional state space by stacking the current two-dimensional slice and multiple future two-dimensional slices; generating multiple rewards for the multiple trajectory decisions based on the recent behavior of the autonomous vehicle; and selecting the final trajectory decision of the autonomous vehicle from the multiple trajectory decisions based on the three-dimensional state space and the multiple rewards. The current two-dimensional slice includes the current self-position of the autonomous vehicle at the current time and multiple current neighboring positions of multiple neighboring vehicles. The multiple future two-dimensional slices include multiple future self-positions of the autonomous vehicle at multiple future time points and multiple future neighboring positions of multiple neighboring vehicles.

[0014] In one or more embodiments of the method, the reinforcement learning is used to select the final trajectory decision.

[0015] In one or more embodiments of the method, the method further includes navigating the autonomous vehicle in response to the final trajectory decision.

[0016] In one or more embodiments of the method, the final trajectory decision is selected by a server computer external to the autonomous vehicle.

[0017] In one or more embodiments of the method, the method further comprises transmitting the plurality of trajectory decisions from the autonomous vehicle to the server computer.

[0018] In one or more embodiments of the method, the method further includes receiving, at the autonomous vehicle, the final trajectory decision from the server computer.

[0019] In one or more embodiments of the method, the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the autonomous vehicle and the plurality of neighboring vehicles.

[0020] A navigation network is provided herein. The navigation network includes a plurality of autonomous vehicles and a server computer. Each corresponding autonomous vehicle in the plurality of autonomous vehicles is configured to: generate a plurality of trajectory decisions at a current time, generate a current two-dimensional slice of a flat space around the corresponding autonomous vehicle, generate a plurality of future two-dimensional slices of the flat space around the corresponding autonomous vehicle by projecting the current two-dimensional slice of the flat space forward in time, generate a three-dimensional state space by stacking the current two-dimensional slice and a plurality of future two-dimensional slices, and generate a plurality of rewards for the plurality of trajectory decisions based on the most recent behavior of the corresponding autonomous vehicle. The server computer communicates with the plurality of autonomous vehicles. The server computer is configured to select a final trajectory decision for each corresponding autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards. The current two-dimensional slice includes a current self-position of the corresponding autonomous vehicle at the current time and a plurality of current adjacent positions of a plurality of adjacent vehicles. The plurality of future two-dimensional slices include a plurality of future self-positions of the corresponding autonomous vehicle at a plurality of future time points and a plurality of future adjacent positions of a plurality of adjacent vehicles.

[0021] In one or more embodiments of the navigation network, the server computer uses reinforcement learning to select the final trajectory decision (advanced actor critic model neural network).

[0022] In one or more embodiments of the navigation network, the server computer implements a high-level actor-evaluator model neural network.

[0023] In one or more embodiments of the navigation network, the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the plurality of autonomous vehicles and the plurality of neighboring vehicles.

[0024] The present invention also includes the following scheme:

[0025] Solution 1. An autonomous vehicle, comprising:

[0026] A plurality of decider modules, coupled to the parser module, and configured to:

[0027] Generate multiple trajectory decisions at the current time,

[0028] generating a current two-dimensional slice of the flat space around the ego vehicle,

[0029] generating a plurality of future two-dimensional slices of the flat space around the autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time, and

[0030] generating a three-dimensional state space by stacking the current two-dimensional slice and the plurality of future two-dimensional slices; and

[0031] a classifier module coupled to the parser module and configured to generate a plurality of rewards for the plurality of trajectory decisions based on recent behavior of the autonomous vehicle, wherein

[0032] The parser module is configured to select a final trajectory decision of the autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards,

[0033] The current two-dimensional slice includes a current self-position of the autonomous vehicle at the current time and a plurality of current neighboring positions of a plurality of neighboring vehicles, and

[0034] The plurality of future two-dimensional slices include a plurality of future self-positions of the autonomous vehicle and a plurality of future neighboring positions of the plurality of neighboring vehicles at a plurality of future time points.

[0035] Option 2. An autonomous vehicle according to Option 1, wherein the parser module uses reinforcement learning to select the final trajectory decision.

[0036] Embodiment 3. The autonomous vehicle of embodiment 1 further comprises a control module coupled to the resolver module and configured to navigate the autonomous vehicle in response to the final trajectory decision.

[0037] Option 4. The autonomous vehicle of Option 1, wherein the parser module is part of a server computer external to the autonomous vehicle.

[0038] Option 5. The autonomous vehicle of Option 4 further comprises a transmitter configured to transmit the plurality of trajectory decisions to the server computer.

[0039] Embodiment 6. The autonomous vehicle according to Embodiment 4 further includes a receiver configured to receive the final trajectory decision from the server computer.

[0040] Option 7. An autonomous vehicle according to Option 1, wherein the multiple future time points represent multiple states in the three-dimensional state space, which summarize multiple movements of the autonomous vehicle and the multiple neighboring vehicles.

[0041] Option 8. The autonomous vehicle according to Option 1 further includes a memory device configured to store lane information, wherein the plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the lane information.

[0042] Option 9. The autonomous vehicle according to Option 1 further includes a memory device configured to store traffic light information, wherein the plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the traffic light information.

[0043] Solution 10. A method for navigation trajectory recognition using reinforcement learning, comprising:

[0044] generating, using the circuit, a plurality of trajectory decisions for the autonomous vehicle at a current time;

[0045] generating a current two-dimensional slice of a flat space surrounding the autonomous vehicle;

[0046] generating a plurality of future two-dimensional slices of the flat space around the autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time;

[0047] generating a three-dimensional state space by stacking the current two-dimensional slice and the plurality of future two-dimensional slices;

[0048] generating a plurality of rewards for the plurality of trajectory decisions based on a recent behavior of the autonomous vehicle; and

[0049] A final trajectory decision of the autonomous vehicle is selected from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards, wherein

[0050] The current two-dimensional slice includes a current self-position of the autonomous vehicle at the current time and a plurality of current neighboring positions of a plurality of neighboring vehicles, and

[0051] The multiple future two-dimensional slices include multiple future self-positions of the autonomous vehicle and multiple future neighboring positions of multiple neighboring vehicles at multiple future time points.

[0052] Option 11. A method according to Option 10, wherein the reinforcement learning is used to select the final trajectory decision.

[0053] Solution 12. The method according to Solution 10 further includes:

[0054] The autonomous vehicle is navigated in response to the final trajectory decision.

[0055] Option 13. The method of Option 10, wherein the final trajectory decision is selected by a server computer external to the autonomous vehicle.

[0056] Scheme 14. The method according to Scheme 13 further includes:

[0057] The plurality of trajectory decisions are transmitted from the autonomous vehicle to the server computer.

[0058] Scheme 15. The method according to Scheme 13 further includes:

[0059] The final trajectory decision is received at the autonomous vehicle from the server computer.

[0060] Embodiment 16. The method according to embodiment 10, wherein the plurality of future time points represent a plurality of states in the three-dimensional state space, which summarize a plurality of movements of the autonomous vehicle and the plurality of neighboring vehicles.

[0061] Solution 17. A navigation network, comprising:

[0062] A plurality of autonomous vehicles, each respective autonomous vehicle being configured to:

[0063] Generate multiple trajectory decisions at the current time,

[0064] Generates the current 2D slice of flat space surrounding the corresponding ego vehicle,

[0065] generating a plurality of future two-dimensional slices of the flat space surrounding the respective autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time,

[0066] generating a three-dimensional state space by stacking the current two-dimensional slice and the plurality of future two-dimensional slices, and

[0067] generating a plurality of rewards for the plurality of trajectory decisions based on recent behaviors of the respective autonomous vehicles; and

[0068] a server computer in communication with the plurality of autonomous vehicles and configured to select a final trajectory decision for each respective autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards, wherein

[0069] The current two-dimensional slice includes a current self-position of the corresponding autonomous vehicle and a plurality of current neighboring positions of a plurality of neighboring vehicles at the current time, and

[0070] The plurality of future two-dimensional slices include a plurality of future self-positions of the corresponding autonomous vehicle and a plurality of future neighboring positions of the plurality of neighboring vehicles at a plurality of future time points.

[0071] Embodiment 18. A navigation network according to Embodiment 17, wherein the server computer uses reinforcement learning to select the final trajectory decision.

[0072] Embodiment 19. A navigation network according to Embodiment 17, wherein the server computer implements a high-level actor-evaluator model neural network.

[0073] Embodiment 20. The navigation network of Embodiment 17, wherein the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the plurality of autonomous vehicles and the plurality of neighboring vehicles.

[0074] The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the disclosure when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a schematic plan view showing the environment of an autonomous vehicle.

[0076] Figure 2 is a schematic diagram of major components of a navigation trajectory system according to one or more exemplary embodiments.

[0077] Figure 3 is a schematic diagram of a computer according to one or more exemplary embodiments.

[0078] Figure 4 is a schematic diagram of a server computer according to one or more exemplary embodiments.

[0079] Figure 5 is a schematic plan view of a decision maker trajectory generation according to one or more exemplary embodiments.

[0080] Figure 6 is a schematic diagram of a transition from a real vehicle perspective to a reinforcement learning state according to one or more exemplary embodiments.

[0081] Figure 7 is a schematic diagram of a current two-dimensional grid according to one or more exemplary embodiments.

[0082] Figure 8 is a schematic diagram of a current two-dimensional grid enhanced according to one or more exemplary embodiments.

[0083] Fig. 9 is a diagram of a visualization of a highway dataset according to one or more exemplary embodiments.

[0084] Fig.10 is a diagram when training on a highway dataset begins according to one or more exemplary embodiments.

[0085] Fig.11 is a diagram of 720 iterations of training according to one or more exemplary embodiments.

[0086] Fig.12 is a diagram of a state space for three time steps according to one or more exemplary embodiments.

[0087] Fig.13 is a graph of rewards obtained in each iteration of training for a single run using a first reward technique according to one or more exemplary embodiments.

[0088] Fig.14 is a graph of rewards obtained in each iteration of a single run of training using a second reward technique according to one or more exemplary embodiments.

[0089] Fig.15 is a diagram of a multi-agent training environment according to one or more exemplary embodiments.

[0090] Fig.16 is a graph of a set of rewards obtained in each iteration of a multi-agent training environment according to one or more exemplary embodiments. DETAILED DESCRIPTION

[0091] Embodiments of the present disclosure provide a navigation network system and / or method for quickly selecting the best steering trajectory for an autonomous vehicle from multiple choices at a given time. An autonomous vehicle is a vehicle that includes sensors for sensing the vehicle's surroundings. The navigation network uses a set of plausible trajectory decisions from multiple decision maker modules as input information. The decision maker module is part of the autonomous vehicle. A neural network is used to train the current state of the autonomous vehicle and the decision maker module so that deep reinforcement learning can be performed using a grading module external to the neural network. The output data from the neural network is the final trajectory decision selected from the input trajectory decisions to be followed by the autonomous vehicle.

[0092] Navigation networks are often implemented with autonomous vehicles and / or highly automated vehicles (HAVs) to quickly select the best steering trajectory from multiple choices at a given time. The selection is made by aggregating the collective experience of multiple vehicles exploring a shared environment. To achieve aggregation, multi-agent reinforcement learning is used to train autonomous agents how to make decisions between a set of possible driving trajectories based on the current state of the agent's vehicle (e.g., the autonomous vehicle). State representations that summarize the motion of other vehicles, as well as map information and traffic information, are also utilized in training. The result is an efficient process for training multiple vehicles simultaneously in a shared environment, which has much faster training time than traditional state and action spaces. In this way, some of the criteria for a decision parser trained based on a variety of experiences are met.

[0093] refer to Figure 1 , a schematic plan view is shown that illustrates the environment of an autonomous vehicle 90. Autonomous vehicle 90 is in communication with server computer 80. In various embodiments, server computer 80 is external to autonomous vehicle 90. Autonomous vehicle 90 generally includes a plurality of wheels 92a-92d and an autonomous driving system 100. Autonomous driving system 100 generally includes a sensor 102, a memory device 104, a computer 106, a power device 108, a receiver 110, and a transmitter 112.

[0094] Signals (e.g., SEN) may be generated by sensors 102 and transmitted to computer 106. Sensor signals SEN may carry information about neighboring vehicles in a portion of the road around autonomous vehicle 90. Signals (e.g., CMD) may be exchanged between computer 106 and power device 108. Command signal CMD may transmit commands to power device 108 to control at least the speed, steering, and braking operations of autonomous vehicle 90. Signals (e.g., DATA) may be exchanged between computer 106 and power device 108. Data signal DATA may transmit data related to the operation of power device 108 between power device 108 and computer 106. Signals (e.g., MEM) may be exchanged between computer 106 and memory device 104. Memory signal MEM may transmit information defining roads in an area used by autonomous vehicle 90. In various embodiments, the information may include lane information and traffic light information of a physical road (or surface). Other types of information may be implemented to meet the design criteria of a particular application. A receive signal (e.g., RX) may be transmitted from server computer 80 to receiver 110. Receive signal RX carries input information about the final trajectory decision of autonomous vehicle 90. Transmit signal (e.g., TX) is generated by transmitter 112 and sent to server computer 80. The transmit signal conveys trajectory candidates for autonomous vehicle 90. In various embodiments, neighboring vehicles also include copies of autonomous driving system 100 that exchange similar signals with server computer 80.

[0095] The server computer 80 can be implemented as one or more central computers. The server computer 80 communicates with the autonomous vehicle 90 via a receiving signal RX and a transmitting signal TX. Communication with other adjacent vehicles can be achieved by similar receiving signals and similar transmitting signals with adjacent vehicles, wherein the adjacent vehicles regard themselves as autonomous vehicles. The server computer 80 is operable to gather the collective experience of multiple vehicles exploring a shared environment. The parser module operating in the server computer 80 provides multi-agent reinforcement learning, which trains how the vehicle as an autonomous agent makes decisions between a set of driving trajectories based on the current state. The state representation of the movement of the general vehicle (or agent) and the map and traffic information are used by the server computer 80 to create an effective process for training multiple vehicles simultaneously in a shared environment. In some embodiments, the server computer 80 can be implemented in the autonomous vehicle 90 to provide the autonomous vehicle 90 with a fully autonomous driving capability.

[0096] Autonomous vehicle 90 may be implemented as an automobile (or car). In various embodiments, autonomous vehicle 90 may include, but is not limited to, a passenger car, a truck, an autonomous driving vehicle, a gas-powered vehicle, an electric vehicle, a hybrid vehicle, and / or a motorcycle. Other types of autonomous vehicles 90 may be implemented to meet the design criteria of a particular application.

[0097] Wheels 92a-92d may be implemented as road wheels. Wheels 92a-92d are generally operable to provide movement of autonomous vehicle 90 with respect to the ground. In various embodiments, each wheel 92a-92d may include a tire mounted on a rim. Wheels 92a-92d may be used to provide traction between autonomous vehicle 90 and the ground on which autonomous vehicle 90 is located.

[0098] The autonomous driving system 100 can be implemented as a power device and associated electronic devices suitable for autonomous, semi-autonomous and / or driver-assisted operation. The autonomous driving system 100 is generally operable to read a road map corresponding to a portion (or section) of the road before (or around) the autonomous vehicle 90, and executes a final trajectory decision for turning, accelerating and / or braking the autonomous vehicle 90. In various embodiments, the scope of the road map can be from tens of feet to hundreds of feet in the direction of travel of the autonomous vehicle 90. The road map can extend before the autonomous vehicle 90 and / or after the autonomous vehicle 90. The scope of the road map can also be from tens of feet wide to about one hundred feet wide. The road map can extend to the left side of the autonomous vehicle 90 and / or the right side of the autonomous vehicle 90.

[0099] Sensor 102 may be implemented as an optical sensor and / or a radio frequency sensor. Sensor 102 is generally operable to sense the movement of adjacent vehicles moving through the portion of the road that autonomous vehicle 90 is about to pass. In various embodiments, sensor 102 may be implemented as a camera configured to observe adjacent vehicles, roads, and obstacles in front of autonomous vehicle 90. In some embodiments, sensor 102 may be implemented as a radar device configured to track the environment. In various embodiments, sensor 102 may further be operable to sense and report detectable lane boundaries depicted on the road.

[0100] The memory device 104 may be implemented as a non-volatile memory device. The memory device 104 is generally operable to store road map information and present it to the computer 106. The road map information may be presented to the computer 106 via a memory signal MEM. The road map information stored in the memory device 104 may be updated from time to time via the memory signal MEM to take into account changes in the road.

[0101] Computer 106 may be implemented as one or more electronic control units. Computer 106 is generally operable to collect sensor information from sensor 102, read a local portion of a road map from memory device 104, receive a trajectory decision from a decision maker module, and execute an optimal approach to navigate autonomous vehicle 90. In some embodiments, computer 106 may transmit individual observations and assumptions of neighboring vehicles to server computer 80 via transmitter 112. The final trajectory decision may be received by computer 106 from server computer 80 via receiver 110, in other embodiments, computer 106 may be implemented as a parser module and thus function independently of server computer 80. Computer 106 may autonomously navigate autonomous vehicle 90, semi-autonomously help navigate autonomous vehicle 90, and / or provide driver assistance in response to the final trajectory decision to traverse the road.

[0102] Power-plant 108 may be implemented as an engine, transmission, and power train of autonomous vehicle 90. Power-plant 108 is generally operable to propel and steer autonomous vehicle 90 in response to one or more commands received from computer 106 in signal CMD. Power-plant 108 may provide operational information back to computer 106 in signal DATA.

[0103] The receiver 110 may be implemented as a wireless receiver. The receiver 110 is operable to receive the final trajectory decision from the server computer 80 via a receive signal RX. The receiver 110 then transmits the final trajectory decision to the computer 106.

[0104] Transmitter 112 may be implemented as a wireless transmitter. Transmitter 112 is operable to transmit a plurality of trajectory decisions and related information of autonomous vehicle 90 determined by computer 106 to server computer 80 in a transmit signal TX.

[0105] refer to Figure 2 , a schematic diagram of an example implementation of several major components of a navigation network 120 is shown according to one or more exemplary embodiments. The navigation network 120 includes four major components: a decision maker module 122 that generates trajectory decisions, a classifier module 124 that grades / rewards trajectory decisions, a parser module 126 that implements a deep reinforcement learning network, and a control module 128 that acts on the final trajectory decision. The parser module 126 is shown as being implemented in the server computer 80. The decision maker module 122 communicates with the parser module 126. The classifier module 124 communicates with the parser module 126. The parser module 126 communicates with the control module 128.

[0106] A trajectory decision signal (e.g., TD) is generated by each decision maker module 122 and transmitted to the resolver module 126. The trajectory decision signal TD carries the trajectory decision. A reward signal (e.g., REWARDS) is generated by the classifier module 124 and transmitted to the resolver module 126. The reward signal REWARDS carries the reward score for the trajectory decision based on the recent behavior of the autonomous vehicle 90. A final trajectory decision signal (e.g., FTD) is generated by the resolver module 126 and transmitted to the control module 128. The final trajectory decision signal FTD transmits the final trajectory decision for the autonomous vehicle 90 to implement.

[0107] At a single time step within the cognitive framework, the decider module 122 creates several candidates for decision making. The trajectory decision is then passed to the parser module 126. With the help of the classifier module 124, the parser module 126 determines which trajectory decision is the most rewarding for the current state of the autonomous vehicle 90, and presents the best decision as the final trajectory decision to the control module 128. The control module 128 thereafter uses the final trajectory decision to control the autonomous vehicle 90 in a fully autonomous mode, a semi-autonomous mode, or to provide advice to the human driver in the case of a highly automated vehicle.

[0108] refer to Figure 3 , a schematic diagram of an example implementation of a computer 106 is shown according to one or more exemplary embodiments. The computer 106 generally includes a classifier module 124, a control module 128, an input circuit 130, and one or more optional directed acyclic graph (DAG) decision maker modules 132 (one is shown for simplicity).

[0109] The input circuit 130 is operable to collect a plurality of trajectory decisions 134a to 134c from a plurality of decision maker modules at a current time. In various embodiments, the trajectory decisions 134a-134c may be received at the input circuit 130 from a directed acyclic graph decision maker module 132 acting as a plurality of local decision maker modules 122. In some embodiments, the trajectory decisions 134a-134c may be received from a plurality of directed acyclic graph decision maker modules 132.

[0110] Directed acyclic graph decision maker module 132 can be implemented as one or more local decision maker modules. Directed acyclic graph decision maker module 132 is operable to generate trajectory decision 134a-134c using sensor information received from sensor 102. As an example, directed acyclic graph decision maker module 132 can generate a directed acyclic graph of three branches. One branch moves in a straight line along the current forward direction of autonomous vehicle 90, and the other two branches switch lanes to the left and right respectively. Then, directed acyclic graph decision maker module 132 distributes pressure field scores to each node of each branch by viewing the distance between the assumed positions of other adjacent vehicles and / or obstacles at the corresponding timestamps of the nodes and each node. The branch with the best average potential field score is selected as trajectory decision (e.g., 134a). Each example of directed acyclic graph decision maker module 132 can be parameterized by speed variables. Speed ​​variables can be the speed of creating branches. For example, higher speeds will make branch nodes more spaced apart than lower speeds. By implementing multiple DAG decider modules 132 with different speed parameters, multiple trajectory decisions may be presented to the input circuitry 130 .

[0111] refer to Figure 4 , a schematic diagram showing an example implementation of a server computer 80 is shown according to one or more exemplary embodiments. The server computer 80 may include a parser module 126. The server computer 80 may be operable to receive information from an autonomous vehicle 90 (e.g., via Figure 1 The server computer 80 is also operable to transmit the final decision to the autonomous vehicle 90 (e.g., via Figure 1 The received signal RX) and adjacent vehicles shown in FIG.

[0112] The parser module 126 may be implemented as a neural network. The parser module 126 is operable to generate a state 140 based on the trajectory decisions received from the vehicle. The parser module 126 may also be operable to generate result data 142, evaluator data 144, and actor data 146 based on the state 140 and the reward generated by the classifier module 124. The final trajectory decision (e.g., 148) among the trajectory decisions (e.g., 134a-134c) received from the vehicle (e.g., 90) may be selected by the parser module 126 and presented to the corresponding vehicle. Therefore, the parser module 126 may train multiple vehicles simultaneously in adjacent environments.

[0113] The parser module 126 implements methods for training and defining states 140, actions, and rewards. The states 140 are used to train a deep reinforcement learning neural network. An example of a suitable neural network is the Advanced Actor-Critic (A2C) model. The Advanced Actor-Critic model uses images as input states (S) and external rewards (R) for training, and then outputs an action (A) to be taken at time (t). The parameters (θ) are trained to maximize the reward. The Advanced Actor-Critic model contains two output layers, one for the policy , one for the value .

[0114] In various embodiments, the parser module 126 implements the high-level actor-evaluator model as a neural network-based reinforcement learning method to learn which decision maker module 122 is optimal for a given state in the driving environment. Neural network-based reinforcement learning methods generally support a large number of agents to be trained in the same environment and have been shown to be effective in many dynamic environments. The following update rules are executed on the neural network:

[0115]

[0116] Advanced Functions Defined as:

[0117]

[0118] Where k is the number of time steps in a training iteration. An asynchronous variant of the high-level actor-critic model, referred to as AC3, can also be implemented in the parser module 126 .

[0119] The structure of the example neural network is a convolutional network with the following layers:

[0120] 1. Input: Dim-40×40×3

[0121] 2. Convolution: Filter-32, Kernel Size-8×8, Stride-4×4, Activation-ReLu

[0122] 3. Convolution: Filter -64, Kernel Size -4×4, Stride -2×2, Activation -ReLu

[0123] 4. Convolution: Filter -64, Kernel Size -3×3, Stride -1×1, Activation -ReLu

[0124] 5. Fully connected: Dim-512, Activation-ReLu

[0125] 6. Strategy output: Dim-4

[0126] 7. Numerical output: Dim-1

[0127] consider Figure 1 and Figure 3 In the illustrated autonomous vehicle 90, the action output of the parser module 126 is an integer indicating the identification of the trajectory decision 134a-134c to be selected. In the parser module 126, each integer corresponds to a unit in the policy output layer. The reward value is a scalar measurement of the value of the learning input state. The reward value can be determined by calculating the distance between the autonomous vehicle 90 and other vehicles or a customized reward function. In various embodiments, the reward value comes from the classifier module 124 that provides the best classification based on the recent behavior of the autonomous vehicle 90. According to the action output of the neural network, the identification of the selected trajectory decision in the trajectory decision 134a to 134c is designated as the final trajectory decision 148. The final trajectory decision 148 is presented to the control module 128 from the parser module 126 through the receiver 110 for operation by the autonomous vehicle 90.

[0128] refer to Figure 5 , a schematic plan view 160 of an example decision maker trajectory generation is shown according to one or more exemplary embodiments. The graph 160 includes a create-and-score trajectory diagram 162 and a select best trajectory graph 164. Each graph 162 and 164 shows an autonomous vehicle 90, wherein a first neighboring vehicle 166a is in front of the autonomous vehicle 90 and in the same lane as the autonomous vehicle 90. Each graph 162 and 164 also shows a second neighboring vehicle 166b in a different lane from the autonomous vehicle 90 and behind it. The generation of candidate trajectory decisions can be implemented by the directed acyclic graph decision maker module 132 within the autonomous vehicle 90.

[0129] The directed acyclic graph decision maker module 132 creates a directed acyclic graph having three trajectory branches for straight driving, left lane switching, and right lane switching. A pressure field score is assigned to each node (circle) according to the distance between the future hypothetical adjacent positions of the node and other adjacent vehicles 166a-166b (commonly referred to as adjacent vehicles 166). In the graph 162, the lowest (best) pressure score 170 can be associated with a node that is unblocked for the autonomous vehicle 90 to occupy. An intermediate pressure score 168 is associated with a node that the autonomous vehicle 90 can navigate through. A strong pressure score 172 can be associated with a node that the autonomous vehicle 90 may reach to a certain extent. The highest (worst) pressure score 174 is associated with a node that the autonomous vehicle 90 should avoid. The branch with the best average potential field score is selected as the best trajectory option (e.g., node 176), as shown in the graph 164.

[0130] Each trajectory decision provided as input information to the input circuit 130 can be generated using a separate strategy. For example, one trajectory decision can be generated by selecting a constant speed, creating branches of possible maneuvers, scoring the branches based on the distance to other neighboring vehicles 166, and selecting the branch with the best score. The best trajectory is used as a single trajectory decision within the set of multiple trajectory decisions 134a-134c collected by the input circuit 130.

[0131] refer to Figure 6 , a schematic diagram of an example transition from a real vehicle perspective to a reinforcement learning state 140 is shown according to one or more exemplary embodiments. Real trajectory 180 shows autonomous vehicle 90 passing through a curved road 182. Each rectangle represents a short (e.g., 400 to 600 milliseconds) period. Egocentric trajectory 184 illustrates curved road 182 as a straight road 186, as seen from a viewpoint in autonomous vehicle 90. Each rectangle represents a short period.

[0132] Based on the real trajectory 180, the current two-dimensional slice is generated by the directed acyclic graph decision module 132. The two-dimensional current slice represents the current state along a specific time range. The two-dimensional current slice contains the current self-position of the autonomous vehicle 90 and the current adjacent position of the neighboring vehicle 166 at the current time. The two-dimensional current slice for each future time step can be projected forward along the specified time range by the directed acyclic graph decision module 132. The future two-dimensional slice contains the future self-position of the autonomous vehicle 90 and the future adjacent position of the neighboring vehicle 166. The two-dimensional slices are stacked together by the directed acyclic graph decision module 132 to form a three-dimensional state space 188. The three-dimensional state space 188 may include a time dimension 190, a flat space dimension 192, and a road width dimension 194.

[0133] The three-dimensional state space 188 can take the form of a three-dimensional tensor. In order to convert the current traffic state into a three-dimensional tensor, future two-dimensional slices of the planar space are stacked, each slice representing a future point in time. In each slice, the hypothetical position of each vehicle is represented as the maximum value (e.g., 255) in a grid that is additionally filled with the minimum value (e.g., 0). The grid is fixed and oriented to the position and heading of the autonomous vehicle 90, where the height and width of each grid square is determined by the width and current speed of the autonomous vehicle 90. Additional information may also be included in the grid, including map features such as lanes and traffic signals. The three-dimensional state space 188 may be transmitted to the parser module 126 for further processing. The additional information provides further context for the neural network to make intelligent choices.

[0134] refer to Figure 7 , a schematic diagram of an example format of a current two-dimensional grid 200 is shown according to one or more exemplary embodiments. Axis 202 of the current two-dimensional grid 200 defines directions extending to the left and right of the autonomous vehicle 90, and axis 204 of the current two-dimensional grid 200 defines directions extending in front of and behind the autonomous vehicle 90. The current two-dimensional grid 200 includes the autonomous vehicle 90 and several (e.g., two) neighboring vehicles 166c-166d. The neighboring vehicle 166c is shown as a right-to-left crossing vehicle in front of the autonomous vehicle 90. The neighboring vehicle 166d is shown as a left-to-right crossing vehicle in front of the autonomous vehicle 90.

[0135] The reinforcement learning state of the system is summarized as a series of two-dimensional grids 210a-210e projected forward into the future. Two-dimensional grid 210a can be at time T = 1 (e.g., current two-dimensional grid 200). Two-dimensional grid 210b can be at time T = 2, etc. Two-dimensional grids 210a-210e are fixed relative to the self-position and heading of autonomous vehicle 90. Hypothetical future neighboring positions of neighboring vehicles 166c-166d in the environment are represented as squares within two-dimensional grids 210a-210e.

[0136] refer to Figure 8 , according to one or more exemplary embodiments, a schematic diagram of an exemplary enhanced current two-dimensional grid 220 is shown. The enhanced current two-dimensional grid 220 may be the current two-dimensional grid 200 ( Figure 7). Some of the additional information may be static (e.g., lanes) and therefore stored in the memory 104. Other additional information may be dynamic (e.g., obstacles) and therefore detected by the sensors 102. The additional information may include, but is not limited to, traffic cones or other obstacles 222, opposing lanes 224, left turn only lanes 226, lanes along the normal route 228, another adjacent vehicle 166e in right turn only lanes 232, and traffic lights and / or traffic signs 234. The additional information may be used to enhance the current two-dimensional grid 220 to enable the parser module 126 to make more informed choices in the trajectory decisions 134a-134c.

[0137] refer to Fig. 9 , a graph 240 of a visualization of a highway dataset is shown according to one or more exemplary embodiments. The graph 240 includes a first dimension 242 (e.g., east-west) and a second dimension 244 (e.g., north-south). A plurality of lane centers 250 are shown in the graph 240. The autonomous vehicle 90 and neighboring vehicles 166 are illustrated as squares. The vehicles 90 / 166 move in a travel direction 246.

[0138] An exemplary training setup for the autonomous driving system 100 is performed using an existing open source implementation of a high-level actor-critic model in Python TensorFlow. This implementation allows custom environments, dynamics, and rewards to be plugged into the existing framework. In this demonstration, a custom environment modeling a lane 250 is created. The environment is populated with vehicle trajectories from a predetermined dataset.

[0139] Single-agent training: Confirm the ability of the parser module 126 to a portion of the predetermined data in a semi-closed loop setting. A semi-closed loop is defined as a main agent (e.g., autonomous vehicle 90) that reacts to other agents (e.g., neighboring vehicles 166), but the other agents do not react to the main agent or to each other. During training, the autonomous vehicle 90 moves using trajectory decisions generated by multiple directed acyclic graph decision maker modules 132, as selected by the parser module 126, while the neighboring vehicles 166 move along their recorded trajectories. The directed acyclic graph decision maker module 132 is directed to generate a path toward the southeast and toward the rightmost lane 250. The default training parameters in the original source code are used, and the learning rate is 7e-2. The configuration parameters used are as follows:

[0140] High-level actor-critic model parameters for single-agent training:

[0141] num_envs: 1,

[0142] unroll_time_steps: 5,

[0143] num_stack: 4,

[0144] num_iterations: 4e6,

[0145] learning_rate: 7e-2,

[0146] reward_discount_factor: 0.99,

[0147] max_to_keep: 4

[0148] refer to Fig.10 , a graph 260 is shown at the beginning of training on an example highway dataset according to one or more exemplary embodiments. The autonomous vehicle 90 is shown in the leftmost lane 250 in the upper left corner. Neighboring vehicles 166 are shown as being dispersed in each lane 250.

[0149] At the beginning of training (e.g., iteration 0), after the high-level actor evaluator model decides which trajectory decisions from which directed acyclic graph decider module 132 to use, candidate trajectory decisions from that directed acyclic graph decider module 132 are plotted, with the stress score (e.g., "+" symbol 262 and circle symbol 264) of each node displayed on the diagram 260. Symbol 262 represents a node where the autonomous vehicle 90 can be located at an appropriate distance from the neighboring vehicle 166. Symbol 264 represents a node where problems may occur when considering the assumed neighboring position of the neighboring vehicle 166.

[0150] refer to Fig.11 , a graph 280 of training at 720 iterations is shown according to one or more exemplary embodiments. Before training (e.g., Fig.10 ), the autonomous vehicle 90 behaves erratically, moving faster than the neighboring vehicle 166 and changing speed frequently. After training 720 iterations, the autonomous vehicle 90 chooses a consistent speed that matches the neighboring vehicle 166 while maintaining an appropriate distance and minimizing the variation between time slices.

[0151] refer to Fig.12 , a graph 300 of an example state space at three time steps is shown according to one or more exemplary embodiments. The state space may have a horizontal dimension 302 and a vertical dimension 304. A slice 306 at an initial time step shows that most of the slice 306 has a value of 0 (white area 312) and the vehicle (black area 314) is represented by a value of 255. A slice 308 at the next time step shows some movement of the vehicle. Slices 310 at subsequent time steps show more movement of the vehicle.

[0152] A Tensor board visualization from high-level actor-critic model training shows the rewards over training. In the demonstration, the reward values ​​are adjusted to a default maximum reward value (e.g., 1) and to a minimum reward value or no reward value (e.g., 0) if there is interference with another vehicle. The reward drops rapidly during these situations, but eventually levels off as the autonomous vehicle 90 learns to drive smoothly. Training is repeated using rewards uniformly sampled from the range of 0 to 1 as a control, in which the behavior remains roughly the same.

[0153] refer to Fig.13 , a graph 320 of example rewards obtained in each iteration of a single run of training using a first reward technique is shown according to one or more exemplary embodiments. Axis 322 of graph 320 shows time. Axis 324 of graph 320 shows rewards ranging from 0 (no reward) to 1 (maximum reward). In the first reward technique, a reward of 1 is given as a default. A reward of 0 is given for interference between the autonomous vehicle 90 and one or more neighboring vehicles 166. Curve 326 shows a smoothed average of the rewards. The dips in curve 326 indicate locations where rewards of 0 are given. Curve 326 generally shows that the number of interferences decreases as the parser module 126 learns the correct policy.

[0154] refer to Fig.14 , a graph 340 of example rewards obtained in each iteration of a single run of training using a second reward technique is shown according to one or more exemplary embodiments. Axis 322 of graph 340 shows time. Axis 324 of graph 340 shows rewards ranging from 0 to 1, where rewards are randomly generated in the range of 0 (no reward) to 1 (maximum reward) in the second reward technique. Curve 342 shows a smoothed average of the rewards. Curve 342 shows that the amount of perturbation is random over time.

[0155] refer to Fig.15 , a graph 360 of a multi-agent training environment according to one or more exemplary embodiments is shown. Graph 360 includes first dimension 242 and second dimension 244. Multiple lane centers 250 are illustrated as lines. Multiple (e.g., five) vehicles 362 are illustrated as squares.

[0156] The scalability of the parser module 126 was tested by extending the environment to include multiple agents trained simultaneously on a shared example of the Next Generation Simulation (NGSIM) highway. The training used a pure closed-loop setting, with no pre-programmed agents and existing agents reacting to each other. To achieve this, a shared Python class was created to track agent positions, assumptions, and renderings of the environment. A separate thread for each agent calculates a separate reward, determines the action at each step, and communicates with a central process for training the key model of the high-level actor. Each thread communicates with the environment class using a ZeroMQ socket. The test showed rough movement in iteration 0, and smoother, more coordinated movement in 1000 iterations. The same training parameters as for a single agent were used, and the "num_envs" parameter was 5.

[0157] refer to Fig.16 , a graph 380 of an exemplary set of rewards obtained in each iteration of a multi-agent training environment is shown according to one or more exemplary embodiments. Axis 322 represents time. Axis 324 represents rewards ranging from 0 to 1. Curves 382, ​​384, 386, and 388 show individual agents training together with other agents in a shared environment. Curves 382, ​​384, 386, and 388 generally show that the agents learn at approximately the same rate.

[0158] In various embodiments of the navigation network, a representation of the current state of the autonomous vehicle is used to make decisions. The navigation network converts vehicle position, traffic status, and map data into a grid-based representation for input into the parser module 126. The navigation network provides a flexible framework for parsing decisions, where the techniques for generating trajectory decision inputs and the classifier module 124 for training the neural network in the parser module 126 can be implemented in various ways. The navigation network identifies the best steering trajectory in an autonomous vehicle that receives a set of seemingly reasonable steering trajectories, transforms the current state of the autonomous vehicle and surrounding agents into a compact grid state, applies the state to a deep reinforcement learning neural network, and outputs the final trajectory decision selected.

[0159] The navigation network increases the convenience of passengers in autonomous vehicles / highly automated vehicles that are controlled using a modular cognitive architecture. For example, an autonomous agent may have several different strategies for generating decisions about future trajectories to take. Choosing the best strategy to use depends on the upcoming situation. The parser module uses deep reinforcement learning in combination with a separate classification module to determine which features of the environment are relevant to picking the best strategy. In addition, the high-level actor-critic neural network has been successfully applied to data from multiple agents experiencing different situations. The network can then utilize data from multiple cars operating in a shared physical space to create dynamic interactions between cars as they learn to coordinate their behavior. In addition, the input space captures the environment in a grid state that includes many contextual cues without adding unnecessary complexity to the problem space. This reduces the resources consumed by training.

[0160] The navigation network exploits trajectory decisions in autonomous driving that follow the laws of physics and are based on maneuvers (e.g., driving in a straight line, switching lanes). Learning these motions from raw data through common techniques requires a lot of time and data. The navigation network simplifies the problem by providing input information as fully formed trajectory decisions based on physical priors. In addition, the neural network can focus training more on contextual cues such as traffic lights and lane restrictions provided in the input state. Instead of an end-to-end approach to learning perceptual recognition of signals and lane types, the navigation network abstracts these tasks to focus on which trajectory decisions best give contextual cues.

[0161] While the best modes for carrying out the disclosure have been described in detail, those familiar with the art to which this disclosure relates will recognize various alternative designs and embodiments for practicing the disclosure within the scope of the appended claims.

Claims

1. An autonomous vehicle comprising: A plurality of decider modules, coupled to the parser module, and configured to: Generate multiple trajectory decisions at the current time, generating a current two-dimensional slice of the flat space around the ego vehicle, generating a plurality of future two-dimensional slices of the flat space around the autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time, and generating a three-dimensional state space by stacking the current two-dimensional slice and the plurality of future two-dimensional slices, wherein the three-dimensional state space includes a time dimension, a flat space dimension, and a road width dimension; and a classifier module coupled to the parser module and configured to generate a plurality of rewards for the plurality of trajectory decisions based on recent behavior of the autonomous vehicle, wherein The parser module is configured to select a final trajectory decision of the autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards, The current two-dimensional slice includes a current self-position of the autonomous vehicle at the current time and a plurality of current neighboring positions of a plurality of neighboring vehicles, and The plurality of future two-dimensional slices include a plurality of future self-positions of the autonomous vehicle and a plurality of future neighboring positions of the plurality of neighboring vehicles at a plurality of future time points; and The autonomous vehicle further includes a control module coupled to the parser module and configured to navigate the autonomous vehicle in response to the final trajectory decision.

2. The autonomous vehicle of claim 1, wherein: The parser module uses reinforcement learning to select the final trajectory decision.

3. The autonomous vehicle of claim 1, wherein: The parser module is part of a server computer external to the autonomous vehicle. 4 . The autonomous vehicle of claim 3 , further comprising a transmitter configured to transmit the plurality of trajectory decisions to the server computer.

5. The autonomous vehicle of claim 3, further comprising a receiver configured to receive the final trajectory decision from the server computer.

6. The autonomous vehicle of claim 1, wherein: The plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the autonomous vehicle and the plurality of neighboring vehicles.

7. The autonomous vehicle of claim 1 , further comprising a memory device configured to store lane information, wherein: The plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the lane information.

8. The autonomous vehicle of claim 1 , further comprising a memory device configured to store traffic light information, wherein: The plurality of decision maker modules are further configured to generate the plurality of trajectory decisions in response to the traffic light information.

9. A method for navigation trajectory recognition using reinforcement learning, comprising: generating, using the circuit, a plurality of trajectory decisions for the autonomous vehicle at a current time; generating a current two-dimensional slice of a flat space surrounding the autonomous vehicle; generating a plurality of future two-dimensional slices of the flat space around the autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time; Generate a three-dimensional state space by stacking the current two-dimensional slice and the multiple future two-dimensional slices, wherein the three-dimensional state space includes a time dimension, a flat space dimension, and a road width dimension; generating a plurality of rewards for the plurality of trajectory decisions based on a recent behavior of the autonomous vehicle; and A final trajectory decision of the autonomous vehicle is selected from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards, wherein The current two-dimensional slice includes a current self-position of the autonomous vehicle at the current time and a plurality of current neighboring positions of a plurality of neighboring vehicles, and The multiple future two-dimensional slices include multiple future self-positions of the autonomous vehicle and multiple future neighboring positions of multiple neighboring vehicles at multiple future time points, The autonomous vehicle is navigated in response to the final trajectory decision.

10. The method of claim 9, wherein the reinforcement learning is used to select the final trajectory decision.

11. The method according to claim 9, wherein: The final trajectory decision is selected by a server computer external to the autonomous vehicle.

12. The method according to claim 11, further comprising: The plurality of trajectory decisions are transmitted from the autonomous vehicle to the server computer.

13. The method according to claim 11, further comprising: The final trajectory decision is received at the autonomous vehicle from the server computer.

14. The method of claim 9, wherein the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the autonomous vehicle and the plurality of neighboring vehicles.

15. A navigation network comprising: A plurality of autonomous vehicles, each respective autonomous vehicle being configured to: Generate multiple trajectory decisions at the current time, Generates the current 2D slice of flat space surrounding the corresponding ego vehicle, generating a plurality of future two-dimensional slices of the flat space around the respective autonomous vehicle by projecting a current two-dimensional slice of the flat space forward in time, generating a three-dimensional state space by stacking the current two-dimensional slice and the plurality of future two-dimensional slices, wherein the three-dimensional state space includes a time dimension, a flat space dimension, and a road width dimension, and generating a plurality of rewards for the plurality of trajectory decisions based on recent behaviors of the respective autonomous vehicles; as well as a server computer in communication with the plurality of autonomous vehicles and configured to select a final trajectory decision for each respective autonomous vehicle from the plurality of trajectory decisions based on the three-dimensional state space and the plurality of rewards, wherein The current two-dimensional slice includes a current self-position of the corresponding autonomous vehicle and a plurality of current neighboring positions of a plurality of neighboring vehicles at the current time, and The plurality of future two-dimensional slices include a plurality of future self-positions of the corresponding autonomous vehicle and a plurality of future neighboring positions of the plurality of neighboring vehicles at a plurality of future time points.

16. The navigation network of claim 15, wherein the server computer uses reinforcement learning to select the final trajectory decision.

17. The navigation network of claim 15, wherein the server computer implements a high-level actor-evaluator model neural network.

18. The navigation network of claim 15, wherein the plurality of future time points represent a plurality of states in the three-dimensional state space that summarize a plurality of movements of the plurality of autonomous vehicles and the plurality of neighboring vehicles.

Images